This isn't you doing nothing wrong and the guy just doesn't like your mitzvos. This is you violating a law that is equally imposed on everyone in the name of public safety. This isn't an "Aiva" thing. This is crapping on the law because you think you're different. Learning Torah is important? 100%. Figure out another way to do it just like everybody else.
It is not equally imposed. Exceptions are carved out for essentials that people can’t live without. By the non-Jews that includes grocery, home improvement centers, and beaches. Religious observance is way down on their list. If the general public was into religious services to the degree they are into their gardening, home improvement projects, and Memorial Day trip to the beach, you can bet that religious services would be prioritized because it would be politically untenable not to do so.
That being said aiva is not a catch all. The first siman in shulchan aruch says one should persist and not be embarrassed from those making fun of him performing mitzvos. Yet we know that one is allowed to put on shoes on tisha be’av when he goes in non Jewish areas where he will be made fun of for going barefoot.
The answer would seem to be that if the non Jews are making fun of the mitzvah being performed one should ignore it, but if the non Jews simply make fun of the Jews lack of shoes not because it is a mitzvah that can be taken into consideration. Similarly, if the aiva is that the non Jews don’t like the fact that Jews are doing mitzvos that is not something that should stop us. On the contrary, that can sometimes be considered shas hashmad requiring more sacrifice to keep doing the mitzvos. However when the not Jews have aiva not because the Jew is doing a mitzvah but because of a legitimate concern, even if it is misplaced and incorrect, that would be aiva and must be taken into account.