If you hate a person, 

you hate something in him that is part of yourself. 

What isn't part of ourselves doesn't disturb us.