Not hard. Challenging. Like missing minyan even though you really might not necessarily kill someone by going just this once
Now that you mention it, maybe this is why the Ari Zal says that you should be mekabel on yourself the Mitzva of Veahavta Lrea'acha komoicha before davening Shachris, so you won't go daven with a minyan and kill someone. (Derech Halotza, obviously).