More often than not, it's the choice of the person being bothered as to whether they are bothered or not.
It reminds what I come across a lot in chinuch. "It's only because you punished him for doing it that it turned into a crime in the first place."
IOW, part of being a mechancech is to be able to not see or hear things. It can get hard for some, but a true mechanech is someone who judges by mind, and not by personal feelings.