Multimodal Medical AI: Beyond the Image

TL;DR

› Foundation models like Merlin combine CT scans, radiology reports, and EHR data to learn richer patient representations than imaging alone allows.

› Merlin was evaluated on 752 tasks and validated on more than 44,000 CT scans across three independent clinical sites and two public datasets.

› Medical images contain predictive information beyond the condition that prompted the scan, including signals for chronic disease development over five years.

› The next generation of medical imaging AI will not be built around images. It will be built around patients.

Introduction

Foundation models have rapidly changed the AI field. Models such as GPT have shown that a single model can learn from massive amounts of data and then be adapted to many different tasks. Medical imaging is now moving in the same direction.

But building a foundation model for medical imaging is not straightforward. CT and MRI scans are three-dimensional and contain hundreds of individual images. Moreover, high-quality training data should also be accompanied by large amounts of clinical information. The question is therefore not simply whether we can build a large model, but whether we can teach it to understand the complex relationship between imaging and the patient behind the image.

A recent paper in Nature takes an important step in this direction. Blankemeier et al. introduce Merlin, a 3D vision-language foundation model trained on abdominal CT scans, radiology reports, and electronic health record diagnosis codes.1 This study is particularly interesting since it shows what becomes possible when imaging data is combined with clinical context.

‍

What Is Merlin?

Most existing medical vision-language models work primarily with 2D images such as chest radiographs. But Merlin is a 3D vision-language model designed specifically for CT. The model was trained using more than 15,000 abdominal CT scans, containing more than 6 million individual images. These scans were paired with more than 6 million tokens from radiology reports and more than 1.8 million diagnosis codes from the electronic health record. Importantly, the authors did not need to manually annotate each CT scan. Instead, Merlin learns from the clinical data that is already generated during routine care.

This is an important shift in how we think about medical AI. Traditional imaging AI often requires a specific dataset with carefully expert-curated labels: pneumothorax, pulmonary embolism, liver lesion, and so on. A foundation model takes a different approach. It tries to learn a more general representation of the image and its relationship to clinical information. The idea is that the same underlying representation can later be used for many different tasks.

‍

One Model, Many Tasks

The authors evaluated Merlin on six different types of tasks, covering 752 individual tasks. Some of these tasks required no additional training. For example, Merlin could perform zero-shot classification of 31 imaging findings and classify 692 different clinical phenotypes. For the 692 phenotypes, Merlin achieved a macro-average AUROC of 0.812. For 102 of the phenotypes, the AUROC exceeded 0.90.

Other tasks required additional adaptation of the model. These included predicting the development of six chronic diseases over five years, generating radiology reports, and segmenting 20 different organs in 3D.

Traditional AI models typically have a narrow focus and are trained for a single task. A traditional AI model might for example be developed specifically to segment the liver. Another model might predict diabetes. Merlin provides a shared representation that can be adapted to all of these tasks. The model therefore becomes less like a single-purpose algorithm and more like a general-purpose imaging engine.

‍

Does the Model Generalize?

Medical imaging AI often performs well on the dataset on which it was developed. The real challenge is what happens when the model is applied to data from another hospital, another scanner, or another patient population. The authors therefore performed external validation.

In addition to an internal test set of 5,137 CT scans, Merlin was evaluated on more than 44,000 CT scans from three independent clinical sites and two public datasets. The external clinical datasets included both abdominal and chest CT scans. Overall, Merlin consistently outperformed the comparison models across the evaluated tasks. This is encouraging because it suggests that the model learned more than institution-specific patterns.

‍

However, external validation does not mean that the problem of generalizability is solved. The external sites were still limited in number, and the training data came primarily from a single health system. We also do not know how Merlin would perform across a much broader range of healthcare systems, geographic regions, scanner vendors, acquisition protocols, or patient populations. As with any foundation model, the diversity of the training data remains critical.

‍

The Importance of Clinical Context

The most interesting aspect of Merlin is the combination of data types. The model does not learn from CT images alone. It learns from CT images together with radiology reports and EHR diagnosis codes. This provides information that is not visible in the image itself. A CT scan might show a particular pattern, but the diagnosis code tells us what happened to the patient. The radiology report provides another layer of clinical interpretation. Together, these data sources create a much richer description of the patient. The authors’ experiments support this idea. Their ablation studies showed that combining EHR data and radiology reports improved performance compared with using EHR data alone.

The question for medical imaging AI now is: Are we really building imaging foundation models, or are we building patient foundation models that happen to include imaging? That distinction may become more and more important.

— Martin Willemink, MD, PhD

‍

From Diagnosis to Prediction

One of the most interesting experiments in the paper was predicting future disease. The researchers used CT scans to predict whether patients who were healthy at the time of imaging would develop chronic diseases within five years. For the multi-disease prediction task, Merlin achieved an AUROC of approximately 0.76. This is not yet a clinical prediction tool, and it should not be interpreted as one. But the experiment demonstrates something important. Medical images contain information about more than the condition that prompted the scan.

A CT obtained for one clinical reason may contain information about body composition, vascular calcification, bone density, organ morphology, or other features associated with future disease. A foundation model may be able to learn some of these patterns without being explicitly trained for each individual prediction task. This opens the door to a different way of thinking about medical imaging: not just as a diagnostic test, but as a rich source of longitudinal information about a patient’s health.

‍

‍What Does This Mean for Medical Imaging Data?

Merlin also highlights a fundamental challenge for the next generation of medical AI. Foundation models need large and diverse datasets. But simply collecting more images is not necessarily enough. The value comes from connecting the images to the information around them. A CT scan without clinical context provides one type of information. A CT scan linked to the radiology report, diagnosis codes, longitudinal medical history, and future outcomes provides something much richer.

The Merlin dataset ultimately released by the authors contains 25,494 abdominal CT scans paired with radiology reports. The model itself was trained on a larger clinical dataset, while the released dataset provides researchers with a resource for further development and evaluation. The study demonstrates what can be achieved when routinely collected clinical data is brought together at scale. It also illustrates why data access, interoperability, privacy, and responsible data sharing will become increasingly important as foundation models develop.

‍

What Are the Limitations?

Despite the impressive results, Merlin is not a universal medical imaging model. First, the model was trained primarily on abdominal CT data. Its ability to generalize to chest CT and other anatomies is encouraging, but it does not mean that the model has learned equally strong representations of every body region. Second, much of the training data came from a single institution. The external validation is a major strength, but broader geographic, demographic, and technical diversity would provide a stronger test of generalizability.

Third, some of the downstream tasks require additional training. The fact that a foundation model can support many tasks does not mean that every task works zero-shot. Finally, the model learns from clinical data that were generated for purposes other than AI training. Diagnosis codes and radiology reports are imperfect representations of reality. They contain omissions, inconsistencies, and biases. A foundation model can learn these patterns as well. A foundation model can only be as representative as the healthcare data from which it learns.

‍

Closing Insight

Merlin is an important step toward foundation models for 3D medical imaging. But the biggest lesson is about data. The future of medical imaging AI will not be built around millions of isolated images with single labels. Instead, it will be built around rich, longitudinal patient-level datasets in which imaging is connected to reports, diagnoses, laboratory results, medications, outcomes, and other clinical information.

‍

The next generation of medical imaging foundation models will not be built around just images. These models will be built around patients.

— Martin Willemink, MD, PhD

‍

What This Means for Real-World Imaging Research

The Merlin study illustrates a shift that is already shaping how imaging data is curated for research. Access to isolated scans is no longer sufficient. What matters is whether imaging is linked to radiology reports, diagnosis codes, treatment records, and longitudinal outcomes.

Segmed curates longitudinal, multimodal imaging datasets linked to clinical, molecular, treatment, and outcomes data across oncology, neurology, and cardiology. This approach reflects the same principle Merlin demonstrates: that the value of a scan depends on the clinical context surrounding it.

‍

Want to explore multimodal imaging datasets for your research or AI development program? Contact Segmed

‍

Frequently Asked Questions - F.A.Q

‍

What is Merlin?

Merlin is a 3D vision-language foundation model designed for CT imaging.1 It was trained on more than 15,000 abdominal CT scans paired with radiology reports and EHR diagnosis codes. Unlike traditional imaging AI, Merlin learns a general representation that can be adapted to many downstream tasks without requiring task-specific labeling.

How many tasks was Merlin evaluated on?

Merlin was evaluated on six task categories covering 752 individual tasks.1 These included zero-shot classification of 31 imaging findings, classification of 692 clinical phenotypes, 5-year chronic disease prediction, radiology report generation, and 3D organ segmentation.

How well did Merlin perform on external data?

Merlin was validated on more than 44,000 CT scans from three independent clinical sites and two public datasets.1 It consistently outperformed comparison models across the evaluated tasks. However, the training data came primarily from a single health system, which limits conclusions about generalizability across broader geographic and technical diversity.

Why does combining imaging with clinical data improve performance?

A CT scan alone shows anatomy and pathology. When linked to radiology reports and EHR diagnosis codes, the same scan provides information about what happened to the patient.1 Merlin’s ablation studies showed that combining EHR data with radiology reports improved performance compared to using EHR data alone.

Can medical imaging predict future disease?

Merlin was used to predict whether patients would develop chronic diseases within five years of a CT scan. For the multi-disease prediction task, the model achieved an AUROC of approximately 0.76.1 This is not yet a clinical prediction tool, but the experiment shows that images contain prognostic information beyond the condition that prompted the scan.

What are the main limitations of Merlin?

Merlin was trained primarily on abdominal CT data from a single institution.1 External validation was performed but geographic, demographic, and technical diversity remains limited. Some downstream tasks require additional fine-tuning. The model also inherits biases and inconsistencies present in the clinical data from which it learned.

What does Merlin mean for the future of medical imaging AI?

The Merlin study demonstrates that the future of medical imaging AI requires rich, longitudinal, patient-level datasets.1 Simply collecting more images is not enough. The value comes from connecting images to radiology reports, diagnosis codes, medications, outcomes, and other clinical data. Foundation models will increasingly be built around patients, not around isolated scans.

References

1. Blankemeier L, et al. Merlin: a computed tomography vision-language foundation model and dataset. Nature. 2026;652:1318–1328.

Related Resources

Koszek A. Medical imaging: an essential component of real-world data. Segmed. Available from: https://segmed.ai/resources/blog/medical-imaging-in-real-world-data

Segmed Experts. Multimodal data pipelines: the new gold standard in pharma research. Available from: https://segmed.ai/resources/blog/multimodal-data-pipelines-in-pharma-research

Willemink M. Fit-for-purpose real-world imaging data: the difference between more data and better data. Available from: https://segmed.ai/resources/blog/fit-for-purpose-real-world-imaging-data-the-difference-between-more-data-and-better-data

‍