Guilty pleasures as an atheist?
The only one I can think of at the moment is a slight affinity for some religious songs. I don't believe any of the content, but some of the themes and melodies are fairly uplifting.
I was listening to some American themes around the 4th of July, and this rendition of the battle hymn of the republic really stood out.
The first part is standard fare, but then they brought out a ringer who really did well.
It's almost enough to fan the embers of belief in something higher.. almost.