I saw another passenger interviewed on the news. He said initially the Dr and his wife offered to give up their seats until they were told that the next flight wasn't until 2:30pm the next day.
That's why he kept saying he had patients to see in the morning. They knew he would miss his appointments with his patients, and they forced him off anyway.