AI Bias & Discrimination Guide 2026: How Training Data Creates Unfair Outcomes

AI bias happens when an artificial intelligence system produces unfair or discriminatory results for certain groups of people. It is not always intentional. Most of the time, the AI does exactly what it was trained to do — and the problem is that it was trained on data that reflects decades of human prejudice. This guide explains how AI bias works, shows you real cases where it caused serious harm, and explains what companies and regulators are doing about it in 2026.

What Is AI Bias?

Think of it this way. You want to hire the best salesperson. You train an AI on 10 years of your company's sales data. Your company historically hired mostly men. The men who were hired mostly had engineering degrees. The AI learns: engineering degree plus male = good salesperson. It was never told to discriminate. It just learned from what it saw.

This is the core problem. AI systems do not understand fairness or justice. They optimise for patterns in data. When that data is the product of an unequal world, the AI amplifies that inequality and presents it as an objective recommendation.

How Training Data Creates Discrimination

Training data is the historical information used to teach an AI model. If that history contains bias, the AI inherits it. There are several mechanisms through which this happens:

Historical Bias

Past human decisions encoded in data contain systematic discrimination. Loan approval records from the 1980s–2000s show lower approval rates for Black and Latino applicants because of discriminatory lending practices. An AI trained on those records learns those approval patterns as "normal."

Representation Bias

When a dataset over- or under-represents certain groups, the AI performs well for over-represented groups and poorly for under-represented ones. A facial recognition system trained on predominantly white male faces will have lower accuracy for women and people of colour — not because the algorithm is intentionally racist, but because those groups made up a small fraction of training examples.

Measurement Bias

The variables used to train an AI may be proxies for protected characteristics. Using a zip code as a variable in a credit scoring model seems neutral — but zip codes in the US and UK closely correlate with race and income. The AI is, in effect, using race as a factor while appearing not to.

Feedback Loops

When an AI system is deployed, its decisions create new data. If a hiring AI rejects candidates from certain universities, those universities produce fewer hires, reinforcing the AI's next training cycle that candidates from those universities are less qualified.

Real Cases of AI Discrimination

Amazon's Hiring Algorithm (2018)

Amazon built an AI to screen CVs for technical roles. The AI penalised CVs that contained the word "women's" (as in "women's chess club") and downranked graduates of all-women's colleges. Amazon discovered the bias and scrapped the tool. The AI had been trained on 10 years of CVs submitted to the company — during which time, most successful hires were men.

Healthcare Priority Algorithm (2019) — US

A landmark study in the journal Science found that a widely used US healthcare algorithm prioritised white patients over Black patients for the same medical conditions. The algorithm used past healthcare spending as a proxy for medical need. Because Black patients had historically received less care — and therefore spent less — they were ranked as having lower needs despite identical or worse health conditions.

COMPAS Recidivism Scores — US Courts

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a risk assessment tool used by US courts to predict criminal recidivism. A 2016 ProPublica investigation found that Black defendants were almost twice as likely as white defendants to be incorrectly flagged as future criminals. White defendants were more often mislabelled as low risk. The tool's creators contested these findings, but it remained one of the most cited cases of algorithmic discrimination in criminal justice.

Apple Card Credit Limits (2019)

Customers reported that Apple Card's Goldman Sachs credit algorithm offered women significantly lower credit limits than men, even when the women had higher credit scores or shared the same household finances as their husbands. Goldman Sachs denied discriminatory intent. New York's Department of Financial Services opened an investigation. No discriminatory intent was proven, but the episode highlighted how opaque algorithms can produce disparate outcomes without anyone inside the company intending them.

Facial Recognition in Law Enforcement

Multiple documented cases exist of Black men in the US being falsely arrested based on incorrect facial recognition matches. Robert Williams in Detroit (2020), Michael Oliver in Louisiana (2023), and Porcha Woodruff in Detroit (2023) were all wrongfully arrested after facial recognition software misidentified them. All three cases involved AI systems with documented accuracy disparities between demographic groups.

Three Sectors Most Affected

Why AI Bias Is Hard to Fix

The Data Problem Is Also a Society Problem

You cannot fix AI bias just by improving the algorithm. If the underlying society is unequal, the training data will reflect that inequality. Solving AI bias requires either collecting better data (which takes decades) or deliberately intervening in what the AI learns and how it weighs different variables.

Fairness Metrics Conflict

There are multiple mathematically valid definitions of "fairness" — and some of them are mutually exclusive. You cannot simultaneously optimise for equal false positive rates across groups and equal positive predictive values. Every fairness choice is a tradeoff, and that tradeoff is a human values decision, not a technical one.

Transparency and the Black Box

Many commercial AI systems — particularly deep learning models — cannot explain why they made a specific decision. A lending AI might reject your application, but it cannot tell you which factors drove that decision in human terms. This makes bias extremely hard to detect and challenge.

What Companies and Regulators Are Doing

EU AI Act (2024)

The EU's AI Act classifies AI systems used in hiring, credit scoring, healthcare, and law enforcement as "high-risk." High-risk systems must undergo conformity assessments, maintain audit trails, undergo bias testing before deployment, and have human oversight mechanisms. Non-compliance can result in fines up to €30 million or 6% of global annual revenue.

US Equal Employment Opportunity Commission (EEOC)

The EEOC issued guidance in 2023 clarifying that AI hiring tools are subject to Title VII of the Civil Rights Act — meaning discriminatory outcomes are illegal regardless of whether the discrimination was intentional. Employers cannot use "the AI decided" as a defence against discrimination claims.

Company Initiatives

• Google — publishes model cards disclosing training data demographics and known limitations for its AI models
• Microsoft — Responsible AI Standard requires bias assessments before deployment and ongoing monitoring
• IBM — provides AI Fairness 360, an open-source toolkit for detecting and reducing bias in AI models
• Hugging Face — requires dataset cards documenting known biases for all datasets on its platform

The gap between company policy and actual implementation remains significant. A 2025 study by the AI Now Institute found that fewer than 30% of companies that published AI ethics principles had implemented auditing mechanisms to verify compliance with those principles.

For businesses using AI for customer-facing decisions, we recommend reading our guide on how AI agents work alongside a framework for bias testing before deployment.

References & Further Reading

• ProPublica — Machine Bias: COMPAS Investigation
• Science — Racial Bias in US Healthcare Algorithm (Obermeyer et al.)
• EEOC — Guidance on AI and Employment Discrimination
• EU AI Act — Official Text and High-Risk AI Requirements