Algorithmic bias examples in criminal justice, hiring, health care and image recognition share one pattern: skewed training data or weak proxies produce worse outcomes for particular groups, visible only when results are checked group by group.
The cases below run from cause to consequence and then to the rules that address them. For the wider regulatory picture, see our ai regulation explained guide.
How bias enters an automated system
Brookings describes algorithmic bias as outcomes that are systematically less favorable to individuals within a particular group when no relevant difference between groups justifies the gap. The same analysis points to unrepresentative or incomplete training data, and to historical inequality, as common origins. A model does not need a discriminatory instruction to produce a discriminatory result. It only needs to learn from records that already carry the imbalance.
Reading the documented cases, bias tends to enter at three points:
- Data collection. If the records describe one group far better than another, or reflect past unequal treatment, the model learns that picture as normal.
- Proxy choice. When the real target, such as illness, is hard to measure, builders substitute something measurable, such as spending. The substitute can carry its own disparities.
- Use in decisions. Once a score reaches a judge, recruiter or clinician, any error in it becomes an error in a real decision about a person.
This sequence is the practical test for any system: ask what the data represents, what the score stands in for, and who acts on it.
Criminal justice and policing
COMPAS and the ProPublica analysis
COMPAS is a recidivism risk algorithm, and it became the best-known case after ProPublica analyzed it in 2017, according to a Santa Clara University library guide. The reporting found that the tool flagged Black defendants as about twice as likely to be misclassified as higher risk. The point for evaluators is that a risk score can look objective while its errors fall unevenly on one group, so error rates need to be examined by group, not just overall.
Risk assessments in the courtroom and predictive policing
Brookings notes that automated risk assessments used by U.S. judges can produce incorrect conclusions, with harsher bail and longer sentences for people of color reported as the effect. Predictive policing software such as PredPol, which the same SCU guide describes, is built on police records. A tool that forecasts where to patrol is only as even-handed as the history of recorded incidents it learned from, so agencies should ask what the training records actually capture before treating a map as neutral.
Judgement for agencies: advisory scores suit settings where a human can see the inputs and override the output. They suit less well where the score quietly becomes the decision.
Hiring and recruiting
Amazon discontinued a recruiting algorithm after discovering gender bias, as the Brookings paper of May 2019 reported. The tool had been trained on resumes submitted over a 10-year period, which were predominantly from white males. That is the clearest documented link between a skewed data set and an unfair outcome: a model fitted to who was hired and who applied in the past will treat that profile as the template for a good candidate.
For employers, the lesson is not that every screening tool fails, but that a vendor’s accuracy claim says little about fairness. Ask what population the training resumes came from, and compare outcomes for men and women, and for different racial groups, before the tool filters anyone out. Where a tool cannot be tested that way, keep it as a sorting aid with human review rather than a gatekeeper.
Health care risk scores
A widely used health risk algorithm showed racial bias because it predicted health care costs rather than illness, according to a Florida State University library guide. Cost works as a stand-in for need only if everyone with the same illness receives the same care. Where that is not the case, Black patients at the same risk score were sicker than their white counterparts, so the score understated their need.
The same guide reports the size of the correction: remedying the disparity would raise the share of Black patients receiving additional help from 17.7% to 46.5%. This case matters beyond medicine because it shows that bias can sit in a perfectly reasonable-looking target variable rather than in any obviously sensitive field. Hospitals and buyers should check what a score actually predicts and compare it against clinical need for each patient group.
Face and image systems
Two image-based cases show how unequal training coverage surfaces in products that were never meant to rank people. In the Beauty.AI contest, which an algorithm judged, nearly all of the 44 winners were white, and only one had dark skin, per the SCU guide. Amazon Rekognition misidentified women as men 19% of the time, and darker-skinned women 31% of the time, according to the same guide.
These errors are less consequential in a beauty contest than in a police or border setting, which is the practical reason to demand subgroup accuracy figures before a face-analysis tool is used in any decision about people. A single headline accuracy number can hide a large gap for one group.
The cases at a glance
| Case | Documented outcome | Planning use |
|---|---|---|
| COMPAS risk scores | ProPublica’s 2017 analysis found Black defendants about twice as likely to be misclassified as higher risk | Request error rates by group, not one overall accuracy figure |
| Beauty.AI contest | Of 44 winners, nearly all white and only one with dark skin | Check how well training images cover every skin tone |
| Amazon Rekognition | 19% of women and 31% of darker-skinned women misidentified as men | Ask for subgroup results before any decision-support use |
| Health risk algorithm | Cost used as a proxy for illness; fixing it would lift the share of Black patients getting extra help from 17.7% to 46.5% | Validate the target variable against clinical need |
Figures come from the Santa Clara University guide and the Florida State University guide.
Rules that address these cases
Brookings recommends updating nondiscrimination and civil rights laws to apply to digital practices. In effect, an automated risk score, résumé filter or health triage tool would be held to the same standards as the human decisions it replaces, which speaks directly to the sentencing, hiring and health care cases above.
A peer-reviewed study in PubMed Central goes further and calls for a transnational independent body with enough power to address AI bias. The argument is that systems and vendors operate across borders, so national rules applied one by one leave gaps.
Two practical readings follow. Where existing anti-discrimination law covers the decision, such as employment or lending, an organisation cannot treat an algorithm as a shield. Where a tool sits outside a clear legal category, audits and public reporting by the organisation itself are the main protection.
The broader ethical debate about how fast to build such systems is set out in Why Some Ethicists Oppose Accelerating AI: 3 Core Arguments and What They Mean (2026) – Technopolicy. Rules aimed at one specific harm from synthetic media are mapped in Which States Make Deepfakes Illegal? The 2026 State-By-State Guide For Consumers And Creators – Technopolicy.
Frequently asked questions
What counts as algorithmic bias?
Brookings frames it as outcomes that are systematically less favorable to a particular group of individuals without a relevant difference that justifies the gap. A single wrong prediction does not qualify. The pattern across a group does.
How widely was the biased health risk tool used?
The Florida State University guide describes tools of this type as used on more than 200 million people in the U.S.
How widely is predictive policing software used?
The Santa Clara University guide says PredPol is used by over 60 police forces in the US.
Does a biased result always mean someone intended it?
No. In the documented cases, the unfair outcomes came from training data, proxy choices or unchecked deployment, rather than from a stated aim to treat groups differently.
