Medical AI Training Relies on Expert Judgment, Not Just Data Labeling
In healthcare AI development, data annotation is not a simple labeling process but a clinical decision made by qualified professionals, such as radiologists or pathologists. The accuracy of AI models depends heavily on the quality and transparency of these expert interpretations, including how disagreements are resolved and documented. Without proper credentialing and traceability, AI systems risk learning biases from inconsistent human judgments.
On October 29, 2026, Peter Leo, a senior consultant at Damco Solutions, published an industry commentary highlighting a critical flaw in healthcare AI development: the misconception that data annotation is merely a mechanical labeling task. The article underscores that clinical annotations—such as those used to train radiology or pathology AI systems—are not simple labels but diagnostic judgments made by qualified professionals like board-certified radiologists and pathologists. These experts interpret complex medical images and texts, often with measurable disagreement; for example, two radiologists may draw different boundaries on the same scan, both being clinically defensible. When such variability is reduced to a single "ground truth" without documentation of who made the call or how disagreements were resolved, AI models risk learning systematic biases rather than true clinical patterns. The article cites research showing that models can achieve 94% agreement with a flawed ground truth not because they are accurate, but because they mirror the idiosyncrasies of individual annotators. This issue extends beyond imaging to clinical text annotation and waveform analysis, where trained specialists are essential.
The background lies in the growing reliance on AI in healthcare diagnostics, where models are expected to match or exceed human performance. However, prior reporting has shown that many AI systems fail under real-world conditions due to poor training data quality.
This commentary could reshape how healthcare organizations approach AI development by elevating data annotation from a technical step to a clinical governance process. If adopted, it may lead to higher-quality AI models that better reflect real-world diagnostic variability and reduce bias, particularly in high-stakes areas like cancer detection or neurological imaging. The emphasis on credentialing and adjudication may also shift market dynamics, favoring specialized clinical data annotation providers over generalist platforms. Comparable to the 2016 case of Google’s DeepMind Health project, where privacy concerns led to regulatory scrutiny and public backlash, this article signals a growing demand for transparency in AI training data—especially when it involves sensitive health information. The implications extend beyond individual institutions: insurers, regulators, and medical device developers may begin requiring auditable annotation workflows as part of approval processes. What to