You are missing out on the mitzvah of tzitzis during that time.
True. But the extra layer is entirely unbearable for me - I tend to be hot even in air conditioning and especially when I'm up at the amud.
I didn't trust myself when it first occurred to me, so I ran it by Rabbi Berkovits and he said it was fine. (That's not a haskama, mind. Not if you know the way RYBs answers questions.)