Is the amount of used RAM really a problem?

Everyone seems to always say that the amount of RAM that's being used doesn't matter because RAM works at its best when used. But in my personal experience (with both computers and phones), the less free RAM I have, the slower my computer/phone works. Would the sluggish use of my computer/phone when the RAM is full my fault or does the whole "used RAM is good for you" argument not apply to every situation?