MedGemma and the Limits of Benchmark Evidence in Imaging AI

TL DR:

• MedGemma is a family of open-weight medical vision-language foundation models from Google Research and Google DeepMind, described in Nature Medicine in 2026.
• It reads medical images and text together, and can be run and fine-tuned on local institutional data.
• On chest radiograph classification from external datasets, MedGemma 4B matched or exceeded previously reported results, though absolute performance remained modest (macro F1 around 50).
• One unblinded radiologist rated 81% of its generated chest X-ray reports as equal or superior to the originals, but only 49% for abnormal studies.
• The models handle 2D images only and downsample inputs to 896 × 896 pixels.
• MedGemma lowers the barrier to building imaging AI, but clinical use still depends on validation with data that represent real patients, scanners and workflows.

Introduction

Access has been a persistent constraint on clinical AI. The most capable multimodal models, which process images and text jointly, are large, expensive to run and usually available only through a vendor's cloud service. These properties make them difficult to adapt to institutional data or to deploy under local privacy requirements.

A study by Google Research and Google DeepMind, published this month in Nature Medicine [1], describes MedGemma, a collection of open medical vision-language foundation models. Here, "open" means that the model weights are publicly released and can be run on institutional infrastructure and retrained on local data. "Vision-language" means that a single model accepts an image together with a text prompt and responds in free text, for example by describing the findings on a chest radiograph. "Foundation model" denotes a general-purpose starting point intended for adaptation to specific tasks; MedGemma has not been evaluated as a clinical product for any indication.

Despite promising results in radiology tasks, the authors are explicit about the limits of this evidence. Performance on established benchmarks, they write, represents

"only the first step towards validating real-world utility."

‍

Research Spotlight

MedGemma is derived from Gemma 3, Google's general-purpose open model, and comprises three variants: two multimodal models with 4 billion and 27 billion parameters and a 27-billion-parameter text-only model. The collection also includes MedSigLIP, a 400-million-parameter medical image encoder. The encoder converts images into representations that the language model can process, and it can also be used independently for image classification and retrieval.

‍

Imaging training data

Radiology accounts for most of the imaging training data. The training set combined public datasets, principally MIMIC-CXR, with internal collections that extend the data previously used for Google's Med-Gemini models.

The non-radiology imaging data comprised approximately 32.5 million histopathology patch–text pairs, 51,049 internal dermatology images and 199,258 retinal fundus images from EyePACS.

Two aspects of data handling are relevant to radiological interpretation. All images are resized to 896 × 896 pixels before input, and the standalone MedSigLIP encoder operates at 448 × 448 pixels. This is below the native matrix of a digital radiograph and may limit sensitivity to small or low-contrast findings. In addition, CT and MRI were presented as individual 2D slices, selected on the basis of report text that specified the location of a finding; the model was not trained on volumetric series.

‍

Chest radiograph classification

Classification was evaluated zero-shot, without task-specific training. Out-of-distribution (OOD) datasets were not used at any stage of training. Because CheXpert and CXR14 originate from institutions other than the source of MIMIC-CXR, results on these sets approximate the model's performance on images from a new site.

Macro F1 was computed over five conditions (atelectasis, cardiomegaly, consolidation, edema and pleural effusion) for CheXpert and MIMIC-CXR, and over three conditions (lung opacity, pneumothorax and fracture) for CXR14. The F1 score for each condition is the harmonic mean of precision (positive predictive value) and recall (sensitivity), on a scale of 0 to 100; macro F1 averages these scores with equal weight per condition, so performance on less frequent findings counts as much as on common ones.

The 15.5–18.1% improvement in chest radiograph classification reported in the abstract corresponds to the difference in macro F1 points between MedGemma 4B and Gemma 3 4B on the two OOD datasets. On both, MedGemma 4B equalled or exceeded the best previously reported results; on the MIMIC-CXR MAIRA test set it remained below Med-PaLM M. The authors note that radiologist-adjudicated labels were used for some test sets, which limits comparison with other publications. In absolute terms, a macro F1 of approximately 50 is modest for findings of this frequency.

Report generation

Report generation was evaluated on MIMIC-CXR with RadGraph F1, which measures agreement between the clinical entities and relations in a generated report and those in the reference report. The pretrained MedGemma 4B scored 29.5 for the findings and impression sections combined. Additional training with reinforcement learning increased the score to 30.3, which the authors report as state of the art for this metric and dataset. For the same sections, MedVersa scored 30.0 and Med-Gemini-2D 24.4. The gain from reinforcement learning was 0.8 points, and because MIMIC-CXR report generation was part of MedGemma's training, this is an in-distribution result.

On the independent ReXrank leaderboard, MedGemma 4B achieved the highest GREEN score, an error-based report metric, on the ReXGradient (0.566) and IU-Xray (0.724) datasets, and intermediate rankings on two further datasets. The authors conclude that relative performance depends on the metric and dataset used. rankings on two further datasets. The authors conclude that relative performance depends on the metric and dataset used.

‍

Expert review of generated reports

The clinical evaluation was performed by a single US board-certified radiologist, who compared the original MIMIC-CXR reports with reports generated by the pretrained MedGemma 4B for 306 studies. Images were reviewed as original DICOM files on a clinical diagnostic viewer. The reviewer was not blinded to report origin.

Overall, 81% of generated reports were rated as likely to result in the same or superior clinical decisions as the original reports. The corresponding figure for the larger Med-Gemini model, 73%, comes from a separate earlier study and is therefore not a direct comparison.

Results differed by study type: 68% of generated reports for normal studies (n = 99) and 49% for abnormal studies (n = 207) were rated equal or superior to the original. The abnormal studies are those in which most of the clinical value lies. The category breakdown in Extended Data Fig. 1 also shows that the 81% includes cases in which both reports supported the correct outcome but the original report was more detailed. The proportion of cases in which the original report was superior and the generated report omitted key findings was larger among abnormal studies.

A qualitative example in the paper shows the same pattern. For one chest radiograph, MedGemma reported a large left pneumothorax with a chest tube in place and a possible right upper lobe nodule. The radiologist's impression was a moderately large left hydropneumothorax with an incompletely expanded left lung, and multiple right lung nodules highly suspicious for metastases. The model identified the principal finding but did not describe the fluid component or the number and likely significance of the nodules.

‍

Anatomical localization

Localization was the weakest imaging capability reported. On the Chest ImaGenome test set, anatomical structure localization reached an intersection over union of 0.16 with MedGemma 27B, compared with 0.08 for Gemma 3 27B and 0.03 for MedGemma 4B. The authors state that there is "substantial room for optimization and improvement" of this capability.

Task-specific adaptation

The main practical claim of the paper is that MedGemma is a more effective starting point for fine-tuning than its general-purpose base model. In three experiments using datasets excluded from training, fine-tuned MedGemma 4B outperformed fine-tuned Gemma 3 4B, with the largest advantage when only 10% of the training data was used. In the radiology experiment, pneumothorax classification on SIIM-ACR (1,068 test radiographs), Gemma 3 4B collapsed to predicting the majority class, whereas MedGemma 4B did not. The authors attribute this at least in part to radiology pretraining.

Because the weights are public, independent groups have evaluated MedGemma on imaging tasks not covered in the paper. On ReXVQA, a chest radiograph question-answering benchmark that includes localization, MedGemma scored 83.84, compared with 77.27 for the best-performing radiology resident. For mammography report generation, a fine-tuned MedGemma 4B outperformed Qwen2.5-VL-7B, including on BI-RADS assessment. In the MedFM-Robust benchmark, MedGemma was the model least affected by image perturbation.

Emerging Signal

The evaluation of medical AI is under increasing scrutiny. The MedGemma authors cite a 2025 Nature Medicine article, "A benchmarking crisis in biomedical machine learning", and call for new, more challenging benchmarks that better reflect real-world utility. As open models make strong baseline performance widely available, performance on representative clinical data is likely to become a more meaningful differentiator than benchmark scores.

Limitations

Several limitations are relevant to imaging applications:

1. Multimodal capabilities are restricted to 2D images. Volumetric CT and MRI were not included in training, and the authors identify their inclusion as an area for future development.

2. CT and MRI training slices were obtained from a single US-based outpatient diagnostic center network.

3. Input images are downsampled to 896 × 896 pixels.

4. The expert evaluation of report generation relied on one unblinded reader and a single public dataset.

5. The authors acknowledge that improvements on older, publicly available benchmarks may partly reflect test data leakage.

The study was funded by Alphabet, and all but two authors are current or former Google employees.

‍

Segmed Lens: What This Means in Practice

When a capable foundation model is freely available, the main constraint on clinical use moves from access to the model to the data used to adapt and validate it.

The authors describe MedGemma as a starting point for task-specific adaptation and identify model errors and distribution bias in training data as issues that require ongoing validation. For imaging, such validation requires datasets that reflect routine practice: multiple institutions and scanner types, abnormal findings at realistic prevalence, longitudinal follow-up and linkage to clinical outcomes. It also requires native volumetric CT and MRI, which the current 2D training does not address.

At Segmed, we see a parallel with the experience of anti-amyloid therapies. Amyloid-related imaging abnormalities (ARIA) characterized on MRI in clinical trials must now be monitored in routine practice, across scanners and protocols that the trials did not represent. Controlled studies establish what is possible, and real-world evidence establishes what really works in practice.
‍

‍

References

1. Sellergren, A., Kazemzadeh, S. et al. An open vision-language model for diverse medical applications. Nature Medicine (2026). https://doi.org/10.1038/s41591-026-04626-w

‍

More perspectives from Segmed

‍

1. Amyloid-Related Imaging Abnormalities (ARIA) in Alzheimer's Disease: From Clinical Trials to Real-World Practice

2. FDA Expectations for Real-World Imaging Data: Building Regulatory-Grade RWE

3. Multimodal Data Pipelines in Pharma Research: Connecting Imaging, Clinical, and Molecular Data

‍