Please help me to rediscover a lost treasure

I've forgotten the name of a Sci-Fi book that I used to read often when I was a teenager. I only recall a few facts from the plot, the books cover and few settings.

Cover: The cover featured a gang standing in front of a baseball diamond trying to catch a baseball (if I remember that correctly). The baseball is really a spaceship.

The aliens in the spaceship are really criminals, very notorious criminals that have a humorous argument about how the people of Earth don't respect themselves. See, they visited us a while back and we called ourselves "Dirt". They came here today and we Call ourselves "Terra" which means "Earth" which means "Dirt". The alien that hates us is mostly a brain/mushroom like creature. Hard to remember.

Basically the space fbi comes to Earth and is arrested by our forces then lied to for a bit. They're afraid of our nuclear weapons, forgot that we are here and the female alients pupils were large enough to allow for her to see through the 2way glass/mirror that normal people can't see through.