It's called gentrification. While not in its classical form all of the demographic shifts in towns and neighborhoods are a form of gentrification. Once the momentum is there, there's nothing to stop it. The big difference for us is that we are much more community dependent than Goyim are. We need local schools to match השקפה, and more importantly we need some basic institutions within WALKING distance. All of which is why frum yidden are seen moving into towns and neighborhoods as a herd. Which others can perceive as threatening.
However, when the gentrification works the other way, as someone living in such a community, I can tell you that the main concern is affordability for the next generation and for like minded people, and second concern is the overall atmosphere of the community.