Everyone’s a little bit racist. Basic cognitive functioning relies on generalizations, after all. Unwarranted initially, in order to get off the ground, and then more or less supported, or rejected, or revised by experience, by evidence. Slightly less basic cognitive functioning requires that we adjust our generalizations in the face of evidence. We adjust our explanatory categories, we fine-tune and narrow in on something that starts to look like likelihood. Or, rather, we do this when there’s sufficient reasons to do so. And those reasons are not just epistemological, they are to a large extent pragmatic. Most of us are lazy thinkers, and all of us have limited time to spend on adjusting categories. We only do when it’s useful for us to do so, or when we have the time and interest to do so at our leisure.
We assign significance to superficial differences like skin color, mode of dress, language, gait, because superficial differences are what signifies internal and explanatory differences when no other information is readily available. (To be fair: superficial difference often DOES correlate with less unimportant things.)
Prejudices are explanatory shortcuts, and quite often they serve us well. Sometimes by being dead-on, otherwise by knitting us closer together with others with similar needs, who jumped to the same unfounded conclusions about the explanatory order of things. The prosocial function of shared false beliefs should not be underestimated. Organized religion springs to mind.
People are, by most accounts, naturally geared towards bigotry, at least under the circumstance of groups competing for resources that seems to have prevailed through a significant part of our formative evolutionary past. The thing to explain is not that we are bigoted, but that we occasionally stop. (Even so, what categories matter, and how to assign value, is, of course, learnt. The tendency is present in the cognitive default state, but needs triggers and particulars to get underway and be filled by content. A person born into a truly egalitarian world would arguably have the tendency to assign significance to the superficial, but lack most of the triggers to do so, and the guidance to do so in any particular manner. Bigots, if they were still to develop, would be of a much more idiosyncratic ilk).
We may be partially excused for being racist, then, as it is in our nature to make a big deal out of differences, and we can often find at least some evidence to back a prejudice up. We are likely to do this, as our prejudices are often at least partly emotional in nature, and we are likely to go looking for confirmation, rather than rejection, of the rationality of those emotions. Emotions too have an evolutionary function, as indications we do well to trust when information as the the value of things is scarce. Awkwardness in the face of the unfamiliar calls for post hoc rationalization (ironically, the pressure towards a racist explanation of that awkwardness increases when there is a reluctance to self-attribute prejudice).
Yet we do blame people for being racists and are presumably right to do so. Why? Because we have every opportunity to revise our first impressions. The evidence as to the irrelevance of these superficial differences is so overwhelming, and the case for the basic equal value of persons is so much stronger than any alternative account, that there is really no excuse if you’re an adult, intelligent person living under non-warlike conditions (people at war, or under extraordinary stress, may sometimes be excused).
The point is that people should be blamed, not merely for harboring racist (and other, avoidable) prejudices, but for failing to revise them in the face of evidence, and for the failure to look carefully enough for evidence that falsifies that prejudice. In many cases, the reason for such failures is nothing more sinister than basic cognitive laziness and self-interest.
There exist, of course, a skeptic safe haven for the bigoted, and it consists of distrust in the sources of evidence. If you believe that, say, the liberal elite has an agenda to mislead concerning the nature and character of the groups you disfavour, you may well protect your prejudices against such challenges. The strategy is fundamentally flawed, as it depends on not adopting skepticism in equal measures to other sources of information, like the sources you use to entrench and confirm your pre-existent prejudice. But the model provides some light when it comes to explaining why some people remain bigoted. It also shows the depth of the problems caused by societal distrust.