Bias in Medical Imaging AI: Diverse Data Is a Start, Not a Fix
Reading time /
5 min
AI Medical Imaging
.png)
TL;DR
- Diverse training data is a precondition for fair medical imaging AI. It does not remove bias on its own.
- Chest X-ray classifiers underdiagnosed underserved patients in four datasets, with higher rates in intersectional subgroups such as Hispanic female patients.1
- Representing each patient group separately does not guarantee performance where those groups intersect.2
- Aggregate accuracy metrics can hide these gaps, so validation must be reported by subgroup and by intersection.
Introduction
Bias in medical imaging AI is often treated as a data volume problem. The logic is simple: add patients from more groups, and the model becomes fair. That logic is half right. Diverse training data is necessary, but it is not sufficient. Radiology research shows that models trained on large, varied datasets can still fail specific patients.1,2 The gap matters most for patients who belong to several underserved groups at once. This post explains why diversity is a precondition rather than a remedy, and what developers need to add on top of it.
Why diverse training data matters for medical imaging AI
A model can only learn from the patients, scanners, and practice patterns it sees in training. When data comes from a narrow set of sources, the model’s performance elsewhere is unknown.
Geography shows how narrow those sources can be. Of US deep learning studies from 2015 to 2019 with identifiable cohorts, 71% used patients from California, Massachusetts, or New York.3 Our post on geographic bias in AI covers that evidence in depth.
The pattern repeats globally. A 2022 PLOS Digital Health review of 7,314 clinical AI papers found that most databases came from the United States (40.8%) and China (13.7%).4 Radiology was the most represented specialty, at 40.4% of papers.4
Narrow sources also teach models the wrong signal. A 2018 PLoS Medicine study tested pneumonia detection models on 158,323 chest radiographs from three hospital systems.5 In three of five natural comparisons, performance on outside hospitals was significantly lower than on the original hospitals.5 The models could also identify which hospital system and department an image came from with very high accuracy.5
Evidence snapshot
Pneumonia models that performed well at home dropped significantly at outside hospitals in three of five comparisons. The same models could tell which hospital system an image came from.5
Why diverse training data alone does not fix bias
A larger and more varied dataset reduces one risk. It does not guarantee equal performance for every patient.
A 2021 Nature Medicine study tested chest X-ray classifiers on three public datasets and on a pooled multi-source dataset.1 The pooled dataset held 707,626 images from 129,819 patients.1 The authors measured underdiagnosis: cases where the model labeled a patient with disease as having no finding.1
The underdiagnosis rate differed across the patient subgroups studied in every dataset.1 In the MIMIC-CXR dataset, female patients, patients aged 0 to 20, and Black, Hispanic, and Medicaid-insured patients had the highest rates.1 The same bias patterns appeared in the pooled dataset.1 Combining sources made the dataset larger and more varied. It did not make the disparities disappear.
Evidence snapshot
A review in the British Journal of Radiology states that adding more minority patients to a training set is not enough to fix bias. Patients belong to several groups at the same time.2
How aggregate metrics hide intersectional bias in AI
Intersectional bias is a performance gap for patients who belong to more than one group, such as female and Hispanic. It can stay invisible when each group is checked separately.
The chest X-ray study shows this directly. In MIMIC-CXR, Hispanic female patients had a higher underdiagnosis rate than white female patients.1 Patients aged 0 to 20 who were also female, Black, or Medicaid-insured had the largest underdiagnosis rates.1
A single headline metric averages across every patient. A small subgroup can perform poorly without moving that average much. Evaluations stratified by race have revealed inequalities in how patients are diagnosed, treated, and billed.6 Reviewers also caution against relying only on mathematical fairness measures, which can miss systemic issues.2
What debiasing techniques can and cannot do
Fairness-aware training methods exist, and they have a role. They cannot add information about patients who are missing from the data.
Adversarial debiasing is a widely cited example. A predictor learns the main task while a second model, the adversary, tries to infer a protected attribute from the predictor’s output.7 The predictor is then trained to make the adversary’s job harder.7 The original paper tested the method on word analogies, census income data, and synthetic data, not on clinical data.7
Simpler fixes carry risks too. Up-sampling and down-sampling patient groups may worsen bias rather than reduce it.2 Detecting the shortcuts that models learn remains an active area of research.2
A framework for reducing bias in medical imaging AI
Diverse data, stratified validation, and ongoing monitoring work together. None of the three replaces the others.
- Build representative training data first. Document demographic composition, geographic origin, scanner and protocol mix, and annotation sources before development begins.
- Validate by subgroup and intersection. Test on data from institutions and populations that contributed nothing to training. Report results by subgroup and at intersections, not only in aggregate.1,2
- Monitor after deployment. Patient populations, protocols, and practice change over time. Keep reporting performance by subgroup once the model is in use.
How Segmed supports representative imaging data
Segmed was founded on the premise that training data quality, diversity, and representativeness limit healthcare AI more than algorithms do. Our network spans 2,800+ healthcare partner sites across five continents. Different institutions serve different populations and run different protocols. We preserve that variation rather than smoothing it away, because it is what a model must handle in deployment. Our clinical team also builds custom cohorts for complex study designs.
Read the full argument in our white paper, Decreasing Bias in Healthcare AI with Segmed. Ready to test your model on the population it is meant to serve? Book a call with our team.
Frequently Asked Questions - F.A.Q.
What causes bias in medical imaging AI?
Bias comes from training and testing data that represent some patients better than others. Models can also learn features of the hospital or scanner instead of the disease.5 Narrow geographic sources make both problems more likely.3
Does more diverse training data eliminate bias in medical imaging AI?
No. Diverse data is a precondition, not a fix. Chest X-ray classifiers still underdiagnosed underserved patients when trained on a pooled dataset of 707,626 images.1 Performance must also be checked by subgroup and at intersections.2
What is intersectional bias in AI?
Intersectional bias is a performance gap for patients who belong to several groups at once. In one chest X-ray study, Hispanic female patients had a higher underdiagnosis rate than white female patients.1 Checking each group separately can miss these gaps.
Why do aggregate accuracy metrics hide bias?
An aggregate metric averages performance across all patients. A small subgroup can perform poorly without changing the average much. Stratified evaluations have revealed inequalities in diagnosis, treatment, and billing that averages did not show.6
Can debiasing algorithms fix a biased training dataset?
Not on their own. Methods such as adversarial debiasing adjust what a model learns from its data, but they cannot add patients who are missing.7 Some rebalancing methods, such as up-sampling and down-sampling, may even worsen bias.2
How should developers test a medical imaging AI model for bias?
Test the model on data from institutions and populations that played no part in training. Report results by sex, race, age, and other relevant groups, and at their intersections.1,2 Continue subgroup reporting after deployment.
References
- Seyyed-Kalantari L, Zhang H, McDermott MBA, Chen IY, Ghassemi M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat Med. 2021;27(12):2176-82. doi:10.1038/s41591-021-01595-0
- Gichoya JW, Thomas K, Celi LA, Safdar N, Banerjee I, Banja JD, et al. AI pitfalls and what not to do: mitigating bias in AI. Br J Radiol. 2023;96(1150):20230023. doi:10.1259/bjr.20230023
- Kaushal A, Altman R, Langlotz C. Geographic distribution of US cohorts used to train deep learning algorithms. JAMA. 2020;324(12):1212-3. doi:10.1001/jama.2020.12067
- Celi LA, Cellini J, Charpignon ML, Dee EC, Dernoncourt F, Eber R, et al. Sources of bias in artificial intelligence that perpetuate healthcare disparities: a global review. PLOS Digit Health. 2022;1(3):e0000022. doi:10.1371/journal.pdig.0000022
- Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 2018;15(11):e1002683. doi:10.1371/journal.pmed.1002683
- Chen RJ, Wang JJ, Williamson DFK, Chen TY, Lipkova J, Lu MY, et al. Algorithmic fairness in artificial intelligence for medicine and healthcare. Nat Biomed Eng. 2023;7(6):719-42.
- Zhang BH, Lemoine B, Mitchell M. Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 2018. p. 335-40.