How much of a difference is there between PS4/XBO versions of games?

I've got an XBO and despite it's flaws I'm pretty happy with it, Forza may be a bit dull and My Kinect has already broken once but it's a nice console. I want to further our relationship and maybe start to buy some third party games, but there's this thing that's always in the back of my mind, a thing that's always nagging and saying "it could be better" and that thing is the PS4.

I've looked at the PS4 many times and went back and forth between the two before launch, at the end of the day I went deep on the XBO for the exclusives that interest me more. Now I'm wondering if it's worth picking up a PS4 so I can play AC and BF and not feel like i'm missing out.

My questions is, If I commit to buying the XBO versions will I feel like I've made a terrible mistake, or are the differences really not that noticeable?