I think the average fan over rates the impact of a manager.
Most of what we have to go by is bullpen management. Girardi was infinitely better at that than his predecessor. While you can always quibble with individual decisions here and there, overall he's been excellent at not overusing or underusing pitchers.
But how can we tell how good he was with players? I'd assume he was pretty good, as the Yankees didn't under perform under him, but who can know for sure? And who is out there that you can be so certain can/will do better?