Is using an iPhone 4 without a case a bad idea?

Having gone through something like my 4th case from general bad case quality, I'm not wondering if I could just get away with not using a case at all.

I'm not one to really drop my phone, or to stick it in my pocket with my keys or anything like that. Is this something that you can generally get away with or is it asking for trouble?