I don't care much for the slur matter of it, but when our way of observing religion is called ultra, its being perceived as extreme, and then we have a problem. The result is less respect and legitimacy for our way of life, from society, from government and from our secular employers. I have no doubt it is damaging. Why should anyone give into for "fanatical extremists".
And yes, that's my issue as well with the anti-vaxxers using religion as an excuse.
I get that no one likes to be labeled or called names, especially by people who know nothing about them. And when you're afraid that name may have negative consequences, it's legitimate to dislike the label. I didn't grow up in a frum community. IME, Chassidim (and to some extent, black hatters) are always looked at as extreme by the outside world. We're in the same category as the Amish. The ones who get to know us form their opinions first-hand, and no media label will change their opinions. The ones who don't get to know us don't need to see "ultra" in the NYT to think what they think about us. People love us and people hate us. This is how it has always been and always will be. I don't believe that we are treated any differently because of that term.
On a personal level, is your approach to Yiddishkeit one of moderation? Do you aim to just be a Jew? Or do you aim higher, more than the bare minimum, a little extra? After 120, would you rather be described as a moderate Yid or an extreme Yid? If they're going to call us names, and they most definitely will, I'd just as well like to own this one.