Nexus 5 without LTE plan (ATT)

I'm planning on getting an unlocked Nexus 5 and using my ATT Sim. I don't really want to upgrade my plan to include LTE, though. Is this likely to cause a problem, or will the N5 just not connect to LTE towers?

Thanks!