Personally, I find it annoying but what can you do? It's their room and they have the right to boot whoever they want and whenever they want for whatever reason. A room owner or the people that are given permissions to boot shouldn't have to be forced to deal with people they dislike in their own room.
If it's for a reason such as prejudice for racial, sexual, and/or gender reasons, then that's when it becomes a problem.