The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation

The Scaffold Effect: When Prompt Framing Masks True Clinical AI Capability

A new preprint from arXiv (2603.28387v2) systematically evaluates 12 open-weight vision-language models (VLMs) on binary classification tasks using clinical neuroimaging data. The core finding is sobering: apparent multimodal performance gains often stem from how prompts are structured—what the authors term the “scaffold effect”—rather than from genuine integration of visual and textual clinical evidence.

The researchers tested models across multiple neuroimaging datasets, varying prompt framing (e.g., direct classification vs. chain-of-thought reasoning, inclusion of clinical context vs. raw image description). They found that certain prompt designs artificially inflated accuracy by exploiting textual cues or surface-level patterns in the data, while more rigorous prompt structures revealed significantly weaker multimodal reasoning. In some cases, models appeared to “succeed” by ignoring image content entirely and relying on biased text priors.

Why This Matters for Clinical AI

This is not an academic quibble. In clinical settings, the difference between a model that genuinely integrates a radiology report with an MRI scan and one that simply memorizes statistical correlations in text is the difference between a useful diagnostic aid and a dangerous illusion. The scaffold effect undermines the validity of many published benchmarks, particularly those that do not control for prompt framing or test for modality collapse—where a model defaults to one input channel.

The study’s methodology is instructive: by systematically ablating prompt components and measuring performance degradation, the authors reveal how much of a model’s “multimodal” capability is actually unimodal shortcut learning. This echoes earlier findings in NLP about spurious correlations, but here the stakes are higher because the input modalities carry different evidentiary weight.

Implications for AI Practitioners

For developers deploying VLMs in healthcare, the lesson is clear: benchmark scores are not trustworthy unless accompanied by rigorous prompt ablation studies. A model that scores 90% on a multimodal clinical task may be doing so by exploiting text biases, not by understanding images. Practitioners should:

• Test for modality collapse by feeding mismatched or corrupted image-text pairs and measuring whether performance drops appropriately.
• Vary prompt framing systematically across evaluation runs, using both minimal and elaborated prompts, to distinguish genuine reasoning from scaffold artifacts.
• Report performance disaggregated by modality—i.e., text-only, image-only, and combined—to surface whether the model is actually integrating information.

The scaffold effect also has implications for model selection: lightweight models that appear competitive on published benchmarks may be more susceptible to prompt artifacts, while larger models with better grounding may show more consistent performance across prompt variations. Until evaluation standards catch up, practitioners should treat any claimed multimodal gain with skepticism unless proven robust to prompt manipulation.

Key Takeaways

• Prompt framing can artificially inflate VLM performance on clinical tasks by up to 20-30%, masking poor multimodal reasoning
• The “scaffold effect” means many published benchmarks may overstate true clinical capability
• Practitioners must conduct prompt ablation and modality-collapse tests before trusting model outputs in healthcare settings
• Evaluation standards for clinical VLMs need to mandate systematic prompt variation and disaggregated modality reporting