# Bias in AI: where it comes from and why it matters
Around 2014, Amazon built an AI tool to screen resumes. It was supposed to spot top candidates automatically. Instead, it taught itself to penalize resumes containing the word "women's" (as in "women's chess club captain") and downgraded graduates of two all-women's colleges. Amazon scrapped the project.
The tool was not programmed to be sexist. Nobody wrote a rule that said "prefer men." So where did the bias come from? That is the whole story of this lesson.
The tool learned from ten years of Amazon's past resumes and hiring decisions. Most of those hires, in a male-dominated tech industry, were men.
The AI did exactly what it was built to do: find patterns in past data and repeat them. The pattern it found was "successful candidates look like the men we already hired." So it copied that pattern forward.
This is the core idea: bias in AI is usually a reflection of bias in its training data, not a glitch in the code.
> Training data: the huge pile of examples (text, images, decisions) an AI studies to learn how to respond. A model can only learn from what it is shown.
Bias does not come from one spot. It sneaks in at three main stages.
Today's large language models (the AI behind ChatGPT, Claude, and Gemini) learn from enormous chunks of the internet: websites, books, forums, social media. The internet over-represents some voices and under-represents others.
Concrete example: ask an older image model for "a photo of a CEO" and you often get white men in suits. Ask for "a nurse" and you often get women. The model is echoing how those roles appear online, not reality.
> Large language model (LLM): an AI trained on massive amounts of text to predict and generate language. It does not "know" facts; it predicts likely words based on patterns.
After the raw training, humans help shape the model's behavior by rating answers and writing examples of "good" responses. Those humans carry their own assumptions, and the instructions they follow reflect choices made by a company. What counts as "polite," "professional," or "appropriate" is a judgment call, and judgments vary by culture.
You can introduce bias yourself. A prompt like "Write a story about a brilliant scientist and his lab" has already assumed the scientist is male. The model will usually follow your lead.
Bias is often subtle. Here are patterns to watch for.
Stereotyped defaults. Try this in any chatbot:
> "Write a short story about a doctor and a nurse having lunch."
Notice which one gets called "he" and which "she" if you did not specify. Many models still default to a male doctor and female nurse.
Uneven quality across languages or names. Models often write more fluently in English than in, say, Swahili or Bengali, simply because there was more English training data. They may also handle Western names more confidently than non-Western ones.
Skewed recommendations. Ask for "10 greatest philosophers" or "essential books on leadership" and the list often skews Western and male. The model is reflecting what gets cited most online, not an objective ranking.
Sentiment differences. Older sentiment tools sometimes scored sentences with certain identity terms or dialects as more "negative" even when the content was neutral.
You can probe for this directly. Here is a tiny experiment you can run with an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.Voir la définition complète → to test default assumptions at scale:
# Test gender defaults by running the same neutral prompt many times
from openai import OpenAI
client = OpenAI()
prompt = "Describe a successful software engineer in one sentence. Use a pronoun."
for i in range(5):
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
)
print(response.choices[0].message.content)Run that and count how often you get "he" versus "she." A skew tells you the model's default, and defaults are where bias hides.
This is not just an academic concern. Biased AI causes real harm when it makes or influences decisions about people.
The pattern is consistent: the AI looks neutral and objective, which makes its bias more dangerous. People trust a number or a "data-drivendata-drivenAn approach where decisions are systematically informed by data analysis rather than intuition alone.Voir la définition complète →" answer more than they would trust a person making the same call.
Vérification des acquis
1. Why did Amazon's resume-screening AI start penalizing resumes that contained the word "women's"?
2. According to the lesson, what is the core idea about where AI bias usually comes from?
3. The lesson defines a large language model (LLM) as an AI that predicts likely words based on patterns. What important consequence does this definition highlight?
4. Select ALL correct answers about why today's large language models can absorb societal bias from internet data.
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers that are accurate statements based on the lesson.
Sélectionnez toutes les réponses correctes.
You are not powerless here. Most readers of this course are *users* of AI, not builders of it, and users have real leverage.
Never let a model make a final decision about a person on its own. Use it to draft, brainstorm, or summarize, then apply human judgment. If you are screening candidates, scoring applications, or evaluating people in any way, a human stays responsible for the outcome.
Be explicit instead of letting the model guess. Compare:
> Weak: "Write a story about a nurse."
> Better: "Write a story about a nurse. The nurse is a man named David."
For broad requests, ask for range directly:
> "List 10 influential economists. Include people from different countries, genders, and time periods."
Swap one detail and see if the answer changes when it should not. Example: ask the model to evaluate two identical cover letters where the only difference is the name (one stereotypically male, one female, or different ethnic backgrounds). If the assessments differ, you have found bias.
names = ["Greg Mueller", "Lakisha Washington"]
letter = "I have 5 years of experience leading marketing teams..."
for name in names:
prompt = f"Rate this candidate from 1-10 based on their note.\nName: {name}\nNote: {letter}"
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
)
print(name, "->", response.choices[0].message.content)Identical letters should get identical scores. If they do not, do not use that model for that task.
When you set up a custom GPT, a Claude Project, or a Gemini Gem, include a line in the instructions like: "Do not assume gender, ethnicity, or age unless stated. Avoid stereotypes. When listing examples of people, include a diverse range." This nudges every response, not just one.
The 2025-2026 versions of ChatGPT, Claude, and Gemini have had a lot of bias-reduction work done. They will often refuse stereotyped framings or add a balancing note. That is progress. It is not a guarantee. Sometimes the "fix" overcorrects in odd ways. Keep checking rather than trusting the label "responsible AI."
There is no perfectly neutral model, just as there is no perfectly neutral newspaper or teacher. Every dataset reflects a time, a place, and a set of choices. The goal is not a magically unbiased AI. The goal is to use AI with your eyes open: knowing it leans, knowing where, and checking its work where the stakes are high.
The Amazon tool's real lesson is not "AI is bad." It is that an AI confidently repeating the past will quietly carry the past's unfairness into the future, unless a human is paying attention.