Foundation Models in Radiology: What Data Do You Actually Need to Fine-Tune Them?

Author:

Jie Wu

Reading time /

6 min

AI Medical Imaging

TL;DR

Foundation models need more than just volume. Data diversity, annotation quality, and modality match matter as much as scale.

Fine-tuning on small, well-curated datasets can outperform training on large, poorly labeled ones; especially with parameter-efficient methods like LoRA.

The biggest bottleneck in radiology AI is not compute. It is access to high-quality, de-identified, well-structured imaging data.

Segmed provides real-world multimodal radiology datasets built for foundation model fine-tuning, without the access barriers.

‍

Why Fine-Tuning Radiology Foundation Models Is a Data Problem

Foundation models are transforming radiology AI. They are large, pre-trained on massive datasets, and designed to adapt to new tasks. But adaptation requires data, the right data.

A lot of teams underestimate what "the right data" actually means. They focus on model architecture. They overlook the fuel.

This blog answers the practical question: what imaging data do you actually need to fine-tune a radiology foundation model?

What Makes Radiology Data Different From General Vision Data

Medical images are not like natural images. They carry physics-based properties unique to each modality. An X-ray encodes tissue density through radiation attenuation. A T2-weighted MRI maps water content in soft tissue. A PET scan shows metabolic activity. These are fundamentally different signals. A model pre-trained on natural images cannot generalize across them without domain-specific adaptation ¹. This is why modality match is so important. Fine-tuning a chest X-ray model on CT data without careful data design produces poor results. Radiology also has a structural data problem. High-quality, real-world imaging datasets are scarce. Privacy regulations restrict sharing. Annotation requires specialist time and expertise ². These constraints make data sourcing one of the hardest parts of building a clinical AI product.

How Much Data Do You Need? It Depends on the Task

There is no single answer. Volume requirements vary by task type, fine-tuning method, and pre-training quality.

The pattern above holds across the literature: the gap between fine-tuning and pre-training is measured in orders of magnitude, not percentages.

The field is moving toward parameter-efficient fine-tuning (PEFT). Methods like LoRA (Low-Rank Adaptation), reduce the number of trainable parameters significantly. They make fine-tuning viable with smaller datasets and lower compute budgets ³.

A 2026 benchmark study tested PEFT methods on chest X-ray prognosis tasks. LoRA and BitFit enabled effective foundation model adaptation even under clinical data constraints like scarcity and class imbalance ³.
‍

The Real Variables in Fine-Tuning Data: Volume, Diversity, and Quality

‍All three matter but not equally

"A small number of labeled samples can be used to fine-tune a foundation model for high accuracy if the pre-trained representations are strong and the fine-tuning data is well-curated." ⁴

Volume is the starting point. Pre-training radiology foundation models typically requires datasets ranging from under 100,000 to multi-million image cohorts¹. But volume alone is not sufficient.

Diversity determines generalization. A model fine-tuned only on one scanner manufacturer, one institution, or one patient population will fail to generalize. Scoping review evidence across 67 radiology FM studies shows that dataset heterogeneity is a consistent gap.¹

Quality is the differentiator at scale. Noisy labels, inconsistent annotation protocols, and missing clinical metadata degrade performance often more than small sample sizes do. For fine-tuning specifically, the evidence points toward curated, modality-matched, well-annotated datasets outperforming raw volume.
‍

What Datasets Are Commonly Used in Radiology FM Research?

MIMIC-CXR: Large chest X-ray dataset with paired radiology reports. Widely used for pre-training and instruction tuning⁵.

CheXpert: Stanford chest X-ray dataset with labels extracted from reports.

UK Biobank: Population-level imaging including cardiac MRI and brain imaging.

NLST: National Lung Screening Trial, used for CT-based models.

Limitation: Public datasets are heavily concentrated on chest X-ray and brain MRI. CT and ultrasound modalities, rare conditions, and pediatric populations are significantly underrepresented.^{1, 2}

This creates a real gap. Most publicly available data does not reflect the diversity of real-world clinical practice. Teams building production radiology AI need access to data that goes beyond what public benchmarks can provide.

‍
The Hidden Risk: Catastrophic Forgetting

Fine-tuning carries a specific risk that is often overlooked: catastrophic forgetting.

When a foundation model is fine-tuned on a small, task-specific dataset, it may lose performance on tasks it previously handled well ². This is not a minor limitation. It affects clinical safety.

The solution is careful fine-tuning design: using PEFT methods that freeze most of the original model weights, and evaluating on held-out tasks beyond the fine-tuning domain. Data quality and representativeness directly shape how much forgetting occurs.

‍

Why Segmed Exists in This Problem Space

Segmed was built to solve the radiology data access problem for clinical AI teams.

We provide de-identified, multimodal imaging datasets purpose-built for AI development across modalities including CT, MRI, X-ray, and PET. Our data is matched to real clinical workflows, includes structured metadata, and is delivered through a compliant, research-ready pipeline.

For teams fine-tuning radiology foundation models, this means:

Modality-matched datasets for your specific downstream task

De-identification and IRB-compatible data access

Metadata structure compatible with instruction tuning and classification workflows

Faster feasibility assessment and cohort creation than starting with new institution-by-institution partnerships
‍

Ready to explore RWiD for your model?
‍Connect with us to discuss your data requirements or submit a project feasibility inquiry to evaluate modality and cohort availability.
‍

‍

Frequently Asked Questions - F.A.Q.

What kind of data do you need to fine-tune a radiology foundation model?

Start with imaging data that matches the target modality, anatomy, population, and clinical task. The labels must also match the intended output: classification may use validated report-derived labels, segmentation requires spatial annotations such as masks, and multimodal instruction tuning requires aligned image-text or image-question-answer pairs. Diversity and label quality matter as much as volume.

‍

How much imaging data is required to fine-tune a radiology AI model?

It varies by task and method. With parameter-efficient fine-tuning (PEFT) approaches like LoRA, effective adaptation is possible with hundreds to low thousands of labeled examples. Pre-training from scratch requires millions of images. Most fine-tuning scenarios fall somewhere in between.
‍

Can foundation models be fine-tuned on small radiology datasets?

Yes, under the right conditions. Strong pre-training representations and parameter-efficient fine-tuning methods like LoRA allow meaningful adaptation with limited labeled data. However, data quality and modality alignment remain critical small poorly labeled datasets will not produce reliable clinical models.
‍

What is catastrophic forgetting in radiology AI fine-tuning?

Catastrophic forgetting occurs when a model fine-tuned on a new task loses performance on tasks it previously handled well. It is a known risk when fine-tuning foundation models on small, domain-specific datasets. Parameter-efficient fine-tuning methods reduce this risk by preserving most of the original model weights.
‍

What is the difference between pre-training and fine-tuning data in radiology?

Pretraining data is usually large and diverse, often reaching hundreds of thousands or millions of images, because its purpose is to learn broadly useful representations. Fine-tuning data is smaller and more task-specific, so label accuracy and relevance to the intended use become especially important.
‍

Why is radiology imaging data so hard to access for AI development?

Radiology data is subject to strict privacy regulations, requires specialist annotation, and is often siloed within hospital systems. These barriers mean that even large AI teams struggle to access the diversity and volume of imaging data needed for robust model development.
‍

What modalities are most underrepresented in public radiology datasets?

Compared with chest X-ray and brain MRI, public datasets provide less coverage of CT, ultrasound, PET, pediatric imaging, rare diseases, and longitudinal multimodal data. Teams working in these areas often need real-world or commercially sourced datasets to supplement public benchmarks.

‍

References

1. Vergara-Richart A, Rafael-Palou X, Fuster-Matanzo A, Iborra Roncales I, Alberich-Bayarri Á, Jiménez-Pastor A. Vision foundation models in radiology: a scoping review of data, methodology, evaluation and clinical translation. arXiv. 2026 Jul. Available from: https://arxiv.org/html/2607.07219v1

2. de Almeida JG, Cerdá Alberich L, Tsakou G, Marias K, Tsiknakis M, Lekadir K, Marti-Bonmati L, Papanikolaou N. Foundation models for radiology — the position of the AI for Health Imaging (AI4HI) network. Insights Imaging. 2025;16:168. Available from: https://link.springer.com/article/10.1186/s13244-025-02056-9

3. Ruffini F, Mulero Ayllón E, Shen L, Soda P, Guarrasi V. Benchmarking foundation models and parameter-efficient fine-tuning for prognosis prediction in medical imaging. Comput Methods Programs Biomed. 2026;275:109196. Available from: https://www.sciencedirect.com/science/article/pii/S016926072500611X

4. Lee D. The application of foundation models in radiology: bridging images, reports, and beyond. J Korean Soc Radiol. 2025;86(6):919-937. doi:10.3348/jksr.2025.0048. Available from: https://pmc.ncbi.nlm.nih.gov/articles/PMC12710298/

5. Paschali M, Chen Z, Blankemeier L, Varma M, Youssef A, Bluethgen C, Langlotz C, Gatidis S, Chaudhari A. Foundation models in radiology: what, how, why, and why not. Radiology. 2025;314:e240597. Available from: https://pubs.rsna.org/doi/10.1148/radiol.240597

6. Ahmad IS, Suleiman RB, Yu T, Lin R, Zhao C, Liao J, Wu B, Xie Y, Liang X, Wang H, Hu Z. Foundation models for X-ray interpretation: a narrative review of current techniques and future perspectives in diagnostic imaging. Quant Imaging Med Surg. 2026;16(4). doi:10.21037/qims-2025-1-2782. Available from: https://qims.amegroups.org/article/view/151394/html

‍