That seems just a bit silly.
Meanwhile I got potched twice that I remember when I was little.
I have no issue if a person feels the need to potch once in a blue moon. Anything more and you're doing something wrong IMO. It shouldn't be necessary.
I think there is actually a Rashi that his actions were based on. That to girls you always teach in a positive manner. To boys, sometimes you got to be tough. I never asked him why he never hit the girls. But the distinction is not silly.
(כה תאמר לבית יעקב ותגיד לבני ישראל" (שמות יט-ג
,רש"י - לבית יעקב - אלו הנשים תאמר להן בלשון רכה
ותגיד לבני ישראל - עונשים ודקדוקין פרש לזכרים