If I knew the person it would not bother me. If I was offended I would let him know.
Presumably, if you knew the person - you'd know if they were a racist or not, and feel comfortable enough to tell them when you thought they stepped over a line. But if a stranger did it - how do I really know?
When I was in High School (about 25-30 years ago), a Black kid about my age gave the nazi salute and said "Heil Hitler" as I passed him by - aside from his ignorance that he'd have been in the concentration camp with me, I had little knowledge of what his true intentions were - was he just making a snarky comment, or if I came back with a (even non-racist) riposte, would that have led him to attack me?
At the same time, have had colleagues occasionally make jokes at my expense, and when it went to far, I'd make it known to them.