I suspect that it goes in all directions... if you grow up in a community consisting exclusively of X, and you're a member of X, you may very well subconsciously inherit some unfair assumptions/beliefs about other groups.

I suspect that it's worse when X is a majority group, but X could also be an isolated minority and the same thing would probably happen.