Hah, same here. I like to get out in the fresh air with my ham gear and inevitably someone asks if I'm talking to Mars, or little green men. 

Only maybe 99% of the time it's when I'm about to break through a pileup with some DX station, too.

I think it's OK, and I don't mind it, but I do try to minimize it. If I can go somewhere without a lot of people, it's just better all around.