That is kind of the point. Not dealing with difficult people has some rather steep penalties. Dealing with them isn't necessarily a lot easier. Either way you are going to pay a price.
The difference between a group that works and one that doesn't is often a matter of how they handle this issue. It is one of the first tests of any leader.
edit spelling