"I don't like children." Is the problem really with them?

In Lithuania, this is a phrase I hear very often. They just say they don't like this entire part of the population. Why? What's wrong with us? Where does such a negative emotion and evaluation come from?