Multi-Model Clinical Validation of an AI-Powered Biomarker Analysis Framework

Summary

Most LLM clinical-validation studies test a single model, leaving vendor dependence unresolved. This study introduces a prompt-based framework that flags eight clinical patterns from laboratory biomarkers and benchmarks five large language models from four providers on 4,018 adults in NHANES 2017–2018, scored against guideline-based ground truth. All five models cleared the clinical-grade threshold (mean F1 ≈ 0.865–0.963); flagship models outperformed economy models, cardiovascular-disease risk was the hardest pattern, and the full benchmark cost about $59. The framework is model-agnostic and supports vendor-independent clinical AI; the authors call for prospective validation before deployment.

Key results

Methods

A prompt-based framework extracts and interprets biomarker panels, flagging eight clinical patterns. Five LLMs from four vendors were run on the NHANES 2017–2018 cohort and scored with F1, sensitivity, specificity and PPV against guideline-based ground truth. Full detail — study design, ground-truth definitions, model configuration, prompt design and statistical analysis — is in the preprint on medRxiv.

Discussion & limitations

The framework performs consistently across vendors, supporting a model-agnostic, vendor-independent approach to clinical AI. This is a retrospective benchmark on a single public cohort; the authors call for prospective validation before any clinical deployment.