Women Not Included: The Bias Was Baked Into the Cake

AI Bias Hiring Technology Ethics

It was half-past three in the morning, and I was, for once, precisely where I intended to be: asleep, or at least attempting it, with a BBC Sounds documentary piped directly into my cranium via a chain of infrastructure so improbable it deserves listing in full: the BBC's studios, the open internet, my home network, a Wi-Fi access point, my phone, Bluetooth, and finally a pair of Soundcore noise-cancelling headphones clamped to my skull like a very patient parasite. This is not an accident of forgetting to switch something off. It is a deliberate, nightly countermeasure. Left unoccupied, my brain treats bedtime as an invitation to keep working, quietly assembling and testing solutions to whatever business problems happen to be queued up that week, with the enthusiasm of a process that has never once been told it's off the clock. A continuous low murmur of somebody else's documentary is, in practice, the only thing that reliably jams that signal long enough for actual sleep to happen. The programme on rotation that night was Women Not Included: Artificial Intelligence, presented by Dr Ella Hubber and journalist Katie Silver, and its thesis is one I have been banging on about since January 2023: artificial intelligence is not neutral, it was never going to be neutral, and pretending otherwise is how you end up with hiring tools, voice assistants, and translation apps that quietly, systematically, fail half the population and disproportionately fail the rest.

The countermeasure, on this particular night, backfired slightly, in that the content was interesting enough to actually register through the fog rather than dissolve into background noise the way most sleep-audio is designed to. I surfaced enough to properly listen, partly because I have spent a meaningful chunk of the last few years building exactly the kind of system the programme was warning about, and I wanted to know whether I'd built it well or built it badly.

 

The Recipe, and Why You Can't Un-Bake the Cake

The programme's central argument is one that should be obvious and yet keeps needing to be restated, loudly, apparently forever: an AI system is not a oracle conjured from first principles, it is a statistical summary of whatever data you fed it, and if that data reflects a biased world, the system will reflect that bias back at you with the confidence of an intern who has never once considered that they might be wrong. Hubber and Silver trace this back to its root, which is the training corpus itself, the vast pile of text, audio, and historical decision-making that machine learning models are shown before anyone lets them near a production system.

Here is the part I found genuinely useful, and the analogy I've been repeating to anyone who'll sit still for it since I heard it: fixing bias in a trained model, after the fact, with fairness constraints and post-hoc adjustments and the occasional stern internal memo about "responsible AI," is a bit like deciding, once the cake has come out of the oven, that you'd rather it hadn't had quite so much butter in it. You can scrape some off the top. You can ice over the worst of it. You cannot, by any act of culinary willpower, remove the butter from the crumb once it has set. The bias is not a bug sitting on top of the model, it is baked into the structure of the thing, distributed through every layer, every weight, every learned association between "nurse" and a pronoun, or "engineer" and a different one. You can mitigate. You cannot fully undo. Which is precisely why the conversation has to happen at the mixing-bowl stage, with the corpus, not at the plating stage, with the finished model.

 

The uncomfortable bit nobody wants printed on the box: the skew isn't a simple tally of who got to write. It's cumulative. Gender bias in the training corpus doesn't just sit next to racial bias, it compounds with it, and both compound again with whatever rhetorical machinery a couple of centuries of persuasive writing, advertising copy, editorial style guides, "authoritative" tone, was built to reward. Stack those three and you don't get addition, you get multiplication: a woman of colour writing on the internet's historical record wasn't simply underrepresented once, or twice, she was underrepresented at every layer the model learns from, including the layer that decides whose voice reads as confident, expert, or worth citing. A model trained on that corpus doesn't invent bias out of malice. It infers the world as documented, and the documentation compounded its unfairness before the first weight was ever trained.

 

Gender-Neutral Languages, Mangled on the Way Through

One of the more quietly damning segments of the programme covered machine translation, specifically what happens when you push a sentence from a language with no grammatical gender, Turkish, Finnish, Hungarian, through a translation model into English. The classic example, which researchers have been citing for the better part of a decade because it remains stubbornly true, is the Turkish sentence "o bir doktor," which is entirely gender-neutral in the source, gets rendered by translation engines as "he is a doctor." Flip the profession to "o bir hemşire" ("nurse") and the pronoun helpfully swaps to "she." The model isn't translating gender, because there is no gender in the source, it is hallucinating gender based on which pronoun most frequently co-occurred with that profession in its training data, and confidently presenting the hallucination as a faithful translation.

This is not a cosmetic quirk. It is the training corpus's occupational stereotypes leaking through a system that was never asked to make a judgement about gender at all, and doing so anyway because statistical association doesn't know the difference between "correlated in the historical record" and "true." Multiply that across every gender-neutral language being machine-translated into English, Spanish, French, at scale, for millions of documents a day, and you have a quiet, continuous, largely invisible act of re-stereotyping the world's professions every time a translation API gets called.

 

ATS Systems, and a Confession About My Own Work

The segment on Applicant Tracking Systems is where the programme stopped being background noise and started making me sit up in the dark like a man who has just remembered he left the oven on. ATS platforms are the software layer that sits between a job applicant and a human recruiter, doing the first pass of filtering, ranking, and in some cases outright rejecting candidates before anyone with a pulse has looked at the CV. The now-notorious case study, which the programme revisits because it remains the clearest public example on record, is Amazon's internal recruiting tool, trained on a decade of the company's own historical hiring data, which taught itself that resumes containing the word "women's," as in "women's chess club captain," were a negative signal, because the historical hiring data it learned from skewed overwhelmingly male. Amazon shelved the tool. The underlying lesson did not get shelved nearly as thoroughly across the industry as it should have.

I mention this because I've spent a meaningful part of the last couple of years building and refining exactly this category of system for achieve.coach1st.co.uk, the career coaching and CV optimisation platform I built for Coach1st. The ATS optimisation routines in that system exist for a very specific reason: they are the countermeasure, not the disease. Job seekers using the platform are, by definition, people trying to get their CV past exactly the kind of filtering logic Amazon had to abandon internally. Coach1st's optimisation engine parses a candidate's CV against a target role, identifies keyword and structural gaps against what the receiving ATS is statistically likely to weight, and rewrites accordingly, so that a genuinely qualified candidate whose CV happens to use different vocabulary, a different formatting convention, or a career history with gaps that a blunt keyword-matcher will misread, doesn't get silently discarded before a human ever sees the name attached to it.

Building that system taught me something I didn't fully appreciate going in, which is that you cannot optimise a CV against an ATS's biases without first understanding, in detail, what those biases actually are, because they are rarely documented and never admitted to by the vendors selling the software. Some ATS platforms penalise employment gaps without regard for cause, which disproportionately affects women who have taken maternity leave and anyone who has taken caring responsibilities more broadly. Some weight extracurricular and professional-association keywords in ways that quietly favour applicants from networks that skew demographically in predictable directions. Some parse non-Western name formats or address structures incorrectly, mangling the applicant's own identifying information before the content of their application is even considered. None of this is intentional in the sense of a developer sitting down and coding in prejudice. All of it is inherited, the same way the translation engine inherits Turkish pronoun stereotypes, from historical hiring data and design defaults that were never audited for who they left behind. The honest position, and the one I try to hold Coach1st's engine to, is that you build the optimisation layer as a bridge across a gap you did not create and cannot single-handedly close, while being loudly, publicly clear that the gap itself is the actual problem and the bridge is a stopgap, not a solution.

 

Compounding: Why the Same Bias Isn't Evenly Distributed

The programme is careful, and correctly so, to push back on any framing of "AI gender bias" as a single, uniform phenomenon that lands equally on all women. It doesn't. Bias in machine learning systems compounds along every axis the training data encodes, and training data encodes race and sexual identity every bit as thoroughly as it encodes gender, usually worse, because there has historically been less corrective attention paid to those axes in the fairness literature. A voice assistant that mishears a woman's voice more often than a man's will misfire more often still on a woman speaking with a regional accent, a non-native speaker's cadence, or a vocal register outside the narrow band the model was actually trained on. A hiring filter that penalises "women's" as a CV keyword compounds with whatever separate penalty it has learned to apply, consciously documented or not, to names, universities, or employment histories statistically associated with race. Stack a third axis, openly queer identity signalled through language, community involvement, or a name change reflected inconsistently across historical records, and you get a compounding effect where the person facing the steepest climb is precisely the person the system was least designed with in mind: a queer woman of colour, filtered by a translation tool that misgenders her, a hiring algorithm that penalises her CV, and a voice assistant that can't reliably parse her own name when she says it aloud.

This is not a hypothetical worst case constructed for rhetorical effect. It is the predictable, mechanical output of stacking independently-biased systems, each individually defensible to its own vendor as "not intentionally discriminatory," none of them evaluated together, on top of a person who has to pass through all of them sequentially just to get a job interview.

 

Voice Recognition, Trained on a Choir That Was Mostly Men

The programme's segment on voice recognition landed with the particular clarity of a statistic you already half-suspected but had never seen stated plainly: early and, in many deployed systems, still-current voice recognition engines were trained overwhelmingly on male voices, a legacy of who was available, funded, and recruited into speech-corpus collection in the decades when those datasets were assembled. The consequence, documented repeatedly by independent researchers over the past several years, is measurably higher word-error rates for women's voices, and higher still for women speaking with accents underrepresented in the training set. Stanford researchers found error rate gaps of close to double digits in percentage terms between male and female speakers on some commercial systems, not because women's voices are objectively harder to parse, whatever the folk explanation might be, but because the model spent its formative training exposure listening mostly to men.

Practically, this means the difference between a voice assistant that understands you the first time and one that makes you repeat yourself, more slowly, more carefully, sometimes in a lower register you have to consciously affect to be understood at all, a small daily tax that falls unevenly and that most of the men designing these systems will simply never personally encounter, and therefore, absent deliberate effort, will never think to fix.

 

What "Fixing It" Actually Requires

The programme ends, as these things generally do, on a note of cautious optimism about better training data, better regulation, and technology built to work fairly for everyone. I share the optimism, with the qualification that better training data is the hard part precisely because of the cake analogy: you cannot retrofit fairness into a corpus after collecting it any more than you can un-bake a cake, which means the fix has to happen at collection time, deliberately, with the kind of institutional patience that most commercial AI timelines are not currently built to accommodate. Regulation helps at the margins, forcing disclosure and audit where market incentives alone would not, but regulation moves at the pace of legislatures while model deployment moves at the pace of a product roadmap, and those two clocks are not presently synchronised in fairness's favour.

What I can vouch for, from the inside of having built one small piece of this ecosystem, is that the honest version of "AI for hiring" has to hold two things true at once: build the tool that helps a real candidate navigate a system that was never built with them in mind, because that candidate needs a job this month, not after the industry gets around to fixing its training corpora, while refusing to let the existence of that tool stand in for the actual fix, which is auditing and rebuilding the underlying filters so the bridge eventually becomes unnecessary. Coach1st's ATS optimisation exists in that first category. It is a crutch, built with care, for a system that shouldn't require one. I'd very much like to put it out of a job.