My problem with the Google Swipe keyboard

I am bringing this up here because

1. I hope you can all help me with this problem.

2. I also wanted to see if this is just my problem.

3. This is one of the reasons I love smaller phones and I know small high end phones is always a interesting topic because they do not exist.

I have about average sized hands so naturally I chose a smaller more compact phone: Moto X.

Even with that smaller phone I still run into this problem and it is why I do not use swipe and why a bigger phone would never work for me.

Take a look at that middle picture below with swipe going on. WHY is the ?123 or the emoji button even press-able while swiping. Every time I use swipe with one hand my palm presses either of those buttons. When that happens the keyboard switches and I lose my word then I either have to juggle my phone to get my thumb down to press the button to go back or use my other hand which defeats the purpose of swipe.

I have been looking for a setting to just disable that button while I am swiping but I have not found anything.

Do I just have dumb hands or is this a non issue for everyone else.