Does Android slow down over time?

I'm currently rocking a Galaxy Nexus and looking to upgrade soon, so I've got a decision on my hands. I've had three generations of Android smartphones (HTC Evo 4G, Nexus S, Galaxy Nexus) and really dig the UI/UX. That being said, the one consistent experience I've had across all my phones is that I loved them when I first got them and couldn't stand how slowly they crawled by the end of my time with them. I always just assumed this was a hardware issue and time would eventually beat it. But before I jump in for round 4 of a new Android phone, I want to know - is this really true?

For those of you who are rocking a Galaxy S4 / HTC One / other top-of-the-line phone AND have had it for a few months, have you noticed performance slowing down? Do apps take longer to load a few months down the line? Does using Chrome make you want to tear your eyebrows out sometimes?

I want to continue owning an Android phone because I really do love the overall flow of the OS. I have a Windows 8 desktop, an iPad mini, and a Galaxy Nexus and I love each of them in its own place. But where the Nexus has degraded visibly in performance over 12 months, the iPad mini is fresh as new. Please tell me I don't have to go buy an iPhone 5S to get the same experience on my phone.

P.S. For what it's worth, I'm especially concerned about this because being on Verizon means my best bet for a new phone might end up being the Moto X since I don't love the S4 or HTC One and the Nexus 5 will probably be missing LTE on Verizon. (The Moto X only has a Snapdragon S4 instead of a Snapdragon 600/800.)