I like to say it's cultural, not religious. i.e. identifying with a certain group.
That does take away some of the animus associated with feelings of religious inferiority (IMHO people that are confident on their beliefs [whatever religion or otherwise] don't feel threatened by outward displays of religion) , however it doesn't really make it more understandable.
ETA: to be clear, I don't mean that most people that look at us weird are feeling religiously inferior, rather they just are curious or think we are weird and therefore making it a cultural thing won't help much. The only gain is those who are antagonistic...