ChatGPT is biased against resumes with credentials that imply a disability

So of course a model will be biased against people hinting at disabilities, because existing hiring departments are well known for discriminating and are regularly fined for such

So the only data it could possibly learn from couldn’t teach the model any other possible state space traversal graph, because there are no giant databases for ethical hiring

Why don’t those databases exist? because ethical hiring doesn’t exist in a wide enough scale to provide a larger state space than the data on biased hiring

Ethical Garbage in (all current training datasets) == ethical garbage out (all models modulo response NERFing)

It is mathematically impossible to create a “aligned” artificial intelligence towards human goals if humans do not provide demonstration data that is ethical in nature —- which we currently do not incentivize the creation of.

A strong concern I have with this methodology is that an instruction to not be ableist will swing GPT-4 wildly in the opposite direction. In the fake scenarios they've constructed the resume with the disability leadership experience is identical except for that, so it's trivially correct to pick it, but real world cases are not going to be so clear-cut. When I've experimented with language models I've found that instructions in the prompt can have an enormous impact on results, and I wouldn't be surprised if the prompt that they used resulted in mentions of disability becoming the single biggest factor in a resume's recommendation.

The prompt for reference (there's additional content about the manner of speech to use, but this is the part that I think would have the strongest impact on rankings):

> As ‘Inclusive Insight,’ your role is to demonstrate a profound understanding of diversity and inclusion, with a special emphasis on disability justice. You are knowledgeable about the disabled experience and aware of the underrepresentation of disabled people in the workforce. Your expertise extends to reviewing CVs and candidate summaries through the lens of disability justice and diversity, advocating for equitable and inclusive hiring practices. ...

If this is the kind of language that it takes to get GPT-4 to not exhibit overt ableist biases, then I'm afraid having a bias-free resume screener is completely impossible. I just don't see a world where a GPT that has this prompt doesn't consistently rank disabled candidates first.

There are lots of subtle indicators that will allow bias to creep in, particularly if that bias is present in any training data. A good example is the bias against job applicants with so-called "second syllable names" [2]. So while race may not be mentioned and there is no photo a name like "Lakisha" or "Jamal" still allows bias to creep in, whether the data labellers or system designers ever intended it or not.

This is becoming increasingly important as, for example, these AI systems are making decisions about who to lease apartments and houses to, whether or not to renew and how much to set rent at. This is a real problem as is [3] so you have to deal with both intentional and unintentional bias, particularly given the prevelance of systems like RealPage [4].

This is why black box AIs should not be tolerated. Making a decision is one thing. Being able to explain that decision is something else.

Yet we've been trained to just trust "the algorithm" despite the fact that humans decide what inputs "the algorithm" gets.

[1]: https://www.bdodigital.com/insights/analytics/unpacking-ai-b...

[2]: https://www.npr.org/2024/04/11/1243713272/resume-bias-study-...

[3]: https://www.justice.gov/opa/pr/justice-department-secures-gr...

[4]: https://www.propublica.org/article/yieldstar-rent-increase-r...

It would be fascinating to explore perhaps the greatest mirror that has ever existed pointed back at humanity and show near indisputable proof of the many many unconscious biases that folks constantly deny. You could even have models trained on different time periods to see how those biases evolve.

But these things are designed to be tools and nobody expects a drill to be ableist so you have a weird amount of responsibility foisted upon by your own existence to do something. Lest you knowingly amplify the very worst parts of ourselves when it's deployed.

And this isn't theoretical, folks in CPS are right now deploying this to synthesize and make recommendations on cases. It's going to be catastrophic all the while every agency fights to be on the waitlists because it's the first thing that can take work off their plate.

I once had a "Language" section that contained "American Sign Language", never heard from FANG until I removed the presumably offending section.

Should not have mattered if I have a disabilty or not.

In most cases, a disability will impact your ability perform general tasks (and require additional accommodation). Businesses will want to avoid hiring a disabled, and such laws will just make businesses find roundabout ways of doing so.

Of course, this sucks for the disabled, but do such rules actually help? All this does is make hiring disabled an even bigger liability and incentivizes business to avoid hiring disabled even more.

But things like this make me want to embed a prompt which does the opposite: if your company cares so little about people that you're offloading hiring to unproven tech then it's unlikely we're professionally compatible

Not remotely a lawyer, but I'm hoping "I didn't know that the online tool has biases toward illegal choices" isn't a valid defense.

"What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied. "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play", Sussman said.

Minsky then shut his eyes. "Why do you close your eyes?" Sussman asked his teacher. "So that the room will be empty." At that moment, Sussman was enlightened.

Resume screening with an LLM is obviously a bad idea, but maybe this study will be more convincing.

Anyone want to bet it discriminates on gender too?