I it based on a gemara . Shiva menudim lashamayim ... one of them is someone who is at a bris and doesnt participate in the seuda. ( It also mentions that the ppl of Yerushalaim would not partake in a seuda if there was a person there of which was not of a certain caliber ( cant really remember the exact loshon) ) So to overcome this it became the minhag not to invite ppl as if the invatation would be considered as if he is there and thus passing up the seuda. As you can see it lacks in logic and halachic bases never-the-less it is a minhag Yisroel