Revisiting Stereotype Threat

I participated as a subject in a research study at Stanford involving race and stereotype threat in the early 2000s. The details are hazy, but the final readout was the distance I put my chair to a group of chairs that students of a particular racial group were supposed to sit. Evidently I put them in a position that was contrary to the effect the researcher was seeking. She intensely asked me a ton of questions about my background and eventually tossed my data point for having lived in a racially diverse area growing up. This wasn't a pre-inclusion criteria, but a possible act of scientific fraud. Huge bummer since there are honest people in every profession, and I imagine a lot of them didn't succeed the way that the fraudsters thrived.

As someone who went through grad studies in the field, and who has met Dr. Inzlicht in person before, I have to say I deeply appreciate his perspective. He has consistently been humble when facing the issues in the field, in ways that call even his own previous research (following the typical practices of the day) into question. The field as a whole has been undergoing a reckoning, but Mickey has been one of the people who has encouraged his fellow researchers not just to wag their fingers at others, but also to look inward and reflect on their own research practices. He has done so by showing humility and acknowledging where his research has fallen short, and that indicates to me a great deal of integrity.

It is sad to see stereotype threat being one of those findings that seems less and less credible. I once worked as a research assistant on a project related to stereotype threat, and I recall the study going through several iterations because it all needed to be just so -- we were testing stereotypes related to women and math, but the effect was expected to be strongest for women who were actually good at math, so it had to be a test that would be difficult enough to challenge them, but not so challenging that we would end up with a floor effect where no one succeeds. In hindsight, it's so easy to see the rationale of "oh, well we didn't find an effect because the test wasn't hard enough, so let's throw it out and try again" being a tool for p-hacking, file drawer effects, etc. But at the time...it seemed completely normal. Because it was.

I'm no longer in the field, but it is genuinely heartening that the field is heading toward more rigour, more attempts to correct the statistical and methodological mistakes, rather than digging in one's heels and prioritizing theory over evidence. But it's a long road, especially when trying to go back and validate past findings in the literature.

Unsurprising. Although this is the first time I can recall reading a psychologist accept culpability for the field’s bad “science” over the last twenty years. Have any of the “anthropology of science” researchers published an explanation of that yet?

>this study (still a preprint) was preregistered (meaning all methods and analyses were specified before the data were collected)

This is the way forward -- preregistered studies. That, together with a promise from the publisher to publish the result regardless of whether the effect is found to be significant.

When you think about it, the incentives for publishing in science have been wrong all along. The future will be different: It will be full of null results, of ideas people had that didn't pan out. But we'll be able to trust those results.

I was a psych major in undergrad, and did an experiment as a riff on stereotype threat and got a small effect. I had the participants solve brain teaser puzzles and the only difference was introducing them as coming from 11th grade or graduate level math. Undergrads did worse when they thought it was graduate level.

Also interesting: p28 (labeled p470) of https://journals.sagepub.com/doi/pdf/10.1177/251524591881022... ("Many Labs 2: Investigating Variation in Replicability Across Samples and Settings") shows the results of attempting to replicate 28 published psych results across many different samples.

Unfortunately, "Stereotype Threat" is not one of the effects they attempted to replicate.

> Let’s be honest: that last sentence was far too generous. Many of us engaged in practices that, in hindsight, were borderline dishonest. We abused experimenter degrees of freedom, engaged in questionable research practices, p-hacked, massaged our data—you pick the euphemism. In contrast, this new replication study followed the most up-to-date best practices in psychological science, eliminating room for flexibility in analysis or results interpretation.

Exploiting researcher degrees of freedom remains unfortunately extremely common. There needs to be some sort of statistical vanguard in the ivory towers enforcing real preregistration and good analysis practices. Strict epistemic discipline is necessary to do real science.

I have seen stereotype threat mentioned in educational contexts - justifying the idea that, for example, it’s important not to make (deliberately or inadvertently) things like coding or engineering seem stereotypically masculine. Usually the recommendations are to ensure that if pictures of participants in classes or extracurricular programs are shown in advertising, diverse groups of people are pictured - or that if connections to popular culture are made in educational materials they’re diverse - for example, don’t make all your example coding projects about Star Wars or football.

Anecdotally, I’ve seen with my own eyes, for example, girls getting really into coding only after seeing it demonstrated by enthusiastic women that they can see as role models in ways they would not see men.

I guess this is a far broader thing than stereotype threat - but I’m sure this larger thing is real. I fear that people who themselves have stereotypes in mind about who ‘should’ be into certain topics will use the demise or deemphasis of stereotype threat to justify not making attempts to attract or be friendly to kids who really could flourish in non-stereotypical fields - to their and society’s detriment.

one of my stances has been that psychology comes up with wacky unreplicable findings because there's no central organizing theories for how 'the mind' works and everything is just very black-box.

there are so many studies showing "X manipulation affects Y outcome", but there's not even a hint of an attempt to explain the mechanisms in a meaningful way (cognitive experiments are usually better, but often still guilty of this).

I also have a high degree of skepticism about most psychology research. I find it frustrating that the author of this piece raises the issue of the imbalance of gender in STEM jobs as a reason to doubt this particular study. There is nothing about this failure-to-replicate that should allow us to conclude that innate differences in math ability underly that imbalance.

I’d love others to read the replication report and explain why I might be wrong?

This is partly the "psychology is the study of college undergrads" problem. The original study was on Stanford students, all of whom have already passed through a very selective filter. That's not a group to extrapolate to the general population. Too many psych studies are done on this convenient population.

It's great seeing the original author admit the problem.

I'm curious what the impact for most people's mindsets will be here? I'm imagining that it relates to what many people consider "talent" with kids? I know that has been an odd trend in some states trying to get rid of "gifted" programs and such. I have largely remained hopeful that that was not nearly as prevalent as online represents.

When I first heard about this stuff 20 years ago, it was being presented as "The differences between groups A and B go away when stereotype threat is removed" which is not what the original paper says. The original paper claims that whatever difference there is between groups A and B, it will be larger when measured under the condition of stereotype threat which I guess is plausible but much less interesting.

Part of the context of that time was that _The Bell Curve_ had been published fairly recently and there was great desire to disprove it and anyone doing that could count on lots of attention and speaking fees. So the grift was to present stereotype threat as this grand solution that could resolve all racial differences.

Link for the claim that the results don't reproduce, without Facebook tracking: https://osf.io/preprints/psyarxiv/qctkp

> When Black students at Stanford University were told that a test was diagnostic of intellectual ability, they performed worse than their white counterparts. However, when this stereotype threat was ostensibly removed—by simply framing the test as a measure of problem-solving rather than intelligence—the performance gap Black and white students nearly vanished.

Just reading this description motivates me to reject the study out of hand. It's not plausible that university-level students responded meaningfully differently to being told "this is a test of problem-solving skill" versus "this is a test of intelligence" because it is commonly understood that problem-solving skill is a major component of intelligence.

>it also became the darling of the political left who now had an answer to prevailing views of group differences held by the political right. This is partly because shortly before stereotype threat took its turn in the spotlight, Charles Murray and Richard Herrnstein published The Bell Curve... the octogenarian Murray is still considered a pariah, shouted down and deplatformed from talks he tries to deliver at respectable colleges to this day.

The characterization of Murray's views in the last several years has been grossly uncharitable and seems entirely disconnected from his actual arguments. It's strange that the book is 30 years old, but has seemed politically relevant for much less time than that.

Executive Summary of Article: "new data now reveal what many of us suspected for at least ten years: stereotype threat does not replicate, and it does not undermine academic performance in the ways we thought."

The Stereotype Threat: "individuals who are part of a negatively stereotyped group can, in certain situations, experience anxiety about confirming those stereotypes, leading paradoxically to underperformance, thus confirming the disparaging stereotype." for example, if you remind a woman of the "Women are bad at math" stereotype, they will perform worse on a math test than if they are not reminded of that stereotype.

Putting on my contrarian hat: the studies he mentions seem to be concerned with whether testing conditions affect stereotype threat effects. Logically, they can't prove or deny the phenomenon's existence, only whether testing conditions result in changes to test outcomes that could track with interventions to reduce stereotype threat. Much of what we know about how behavioral effects of identity comes with the understanding that the latter is something people carry with them, regardless of local or recent events. If there's a problem with stereotype threat as a concept, it's that it's positioned as a superficial effector that can be manipulated easily, rather than the surface level manifestation of complex interactions between self-identity, personal values, and cultural expectations. Based on the author's disclosure about his PhD thesis, he seems to be someone who capitalized on the former characterization, so of course he throws the baby out with the bathwater when it no longer works for those purposes. We might be looking at an ass-covering write-up.

>Let’s play “Find the Lebowski quotes game” again!

So, yeah, I find this a deeply unserious blog post.