Yes, I think it is a "Chabad thing," as you call it, to be sensitive about cholov Yisroel. There is definitely an aspect of that. But there is more to it.
Cholov Yisroel became the red-headed stepchild of halachos. It get's so little respect in the frum community, which is ironic because cholov Yisroel is something you put into your body and which becomes your very organs and cells. Non-cholov Yisroel clogs up the fine machinery that is the Yiddisher mind and heart and makes us unreceptive to ruchnius. So, yeah, in that sense, it is the source of many other ills. People can laugh and scoff at this but when someone loses their sensitivity to other mitzvos, it often can be traced back to maachalos asuros and their numbing effect. Then it becomes a vicious cycle.
TL;DR if frum people today would treat CY with at least as much respect as other halachos, I probably would never mention it.