Microsoft's diagnostic AI outperformed individual doctors in a Science paper. Its cheapest setting scored 79.6% at an estimated testing cost of $1,303 per case.
Microsoft says its diagnostic AI beats doctors "at half or even one third of the cost", in its own post. The paper in Science, published on 22 October, shows the cost depends on the setting, from $1,303 to $8,945 a case.
The paper scores the system at 83.2% when it runs on GPT-5, OpenAI's model. That run averages $5,431 of tests per case. The cheapest run scores 79.6% at about a quarter of that bill.
Generalist doctors scored 36.1% open book and 22.3% closed book. Open book means they could use reference tools. They were active primary care and hospital physicians working from text alone.
Open book, doctors averaged $4,725 a case and 18.1 minutes. Closed book they averaged $3,049 and 11.9 minutes. The top DxO run costs more than both.
How the benchmark worked
The test used published case records, not patients. The authors turned 304 conference cases curated by academic clinicians, from the New England Journal of Medicine, into text exchanges. They added 325 emergency admissions from a hospital database called MIMIC-IV-ED.
A language model plays gatekeeper and answers each request for a history, an examination or a test. Where the record lacks a finding, it invents a neutral result. A second model scores each diagnosis on a five level rubric, and a 4 or 5 counts as correct.
The system is called DxO in the paper and MAI-DxO in Microsoft's blog. It is an orchestration layer that asks questions and orders tests in sequence. Every author is a Microsoft employee holding company equity. Eight of them are inventors on a Microsoft patent application for an automated evaluation framework for sequential diagnosis. The paper reports no outside funding.
The headline comparison uses different case sets for the two sides. The doctors were scored on 56 cases held out from development, and the paper's chart marks them "test set only". The main text does not say which cases the 83.2% covers, so the 47 point gap is indicative.
The paper says DxO set a new Pareto frontier, meaning it beat the physicians at every cost or accuracy level. Microsoft's post goes further. It says no other human or AI approach was both more accurate and less costly.
Testing costs are estimated from the investigations each side requests. Prices come from a major health system's chargemaster, its price schedule. History and examination questions count as a fixed visit cost.
The trade-off between accuracy and testing cost
Lower-cost settings retain much of the accuracy. The paper says accuracy dips slightly as testing costs are cut. One setting scores 80.9% for $2,241.
The cheapest setting scores 79.6%, while the "diagnostic excellence" setting scores 84.5% for $8,945. The highest accuracy and the lowest cost come from different runs.
DxO also works differently from doctors. Physicians asked a mean of 5.6 questions before ordering their first test. DxO asked a mean of 25.5 questions before its first test on the same cases.
The paper says DxO tended to ask more questions and order fewer tests, especially under budget constraints. Even the best single physician, at 51.8%, was outperformed on both accuracy and cost.
The two sides also err differently. DxO's mistakes were usually a clinically adjacent answer at the wrong level of specificity. Physicians' wrong diagnoses tended to sit further from the reference diagnosis.
Results on emergency admissions
Emergency cases are closer to routine care. The top setting scores 80.3% at $4,979 a case, against 46.8% at $3,400 for physicians.
Budget settings score 73.2% for $2,634 and 69.5% for $2,037. Both beat the physicians on accuracy and on cost.
Physician accuracy rose from the case set to the emergency set, while DxO stayed comparable on both. The authors write that they "had expected to see a smaller advantage" for DxO here. They hypothesise that models train on text spanning rare diseases and atypical presentations, a breadth no individual physician could master.
The broad result follows earlier findings. A June 2025 Microsoft post said the system correctly diagnoses "up to 85%" of New England Journal cases. That was more than four times the rate of a group of experienced physicians. The best run paired the orchestrator with OpenAI's o3 model and scored 85.5%.
Science adds peer review and the emergency set. It also adds a repeat of the full evaluation with GPT-5.5, run without changing prompts or orchestration. Microsoft is releasing the benchmark, gatekeeper, judge and cost code for select cases as open source, so outsiders can rerun the test.
The authors say DxO was developed when GPT-4.1 was the strongest model available and evaluated with GPT-5. Across all the models they tested, orchestration added 7.9 points on average on the case set over a plain prompt. On the emergency set it added 4.4.
The model without orchestration scores close to the headline result. TechCrunch reports that GPT-5 with a short, basic prompt scored 78.9% on the complex cases at about $8,400 a case. DxO cut about $3,000 from that bill and added about four points.
With GPT-5.5 the advantage over the baseline prompt grew. The authors say this suggests "that improvements in model capability do not necessarily diminish the benefits of careful orchestration".
Why benchmark results do not establish clinical superiority
Doctors work in teams, and the benchmark uses idealised conditions. The authors acknowledge these limitations. They write that the comparisons "do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice".
The test sessions also lacked the noise of real histories, where patients omit or forget information. The authors add that routine practice "appropriately prioritizes avoiding missed diagnoses and ensuring patient safety over explicit optimization of cumulative testing costs".
The paper does not report cost per correct diagnosis. Dividing each DxO average bill by its accuracy gives about $1,600 for the 79.6% setting and $6,500 for the 83.2% setting. The last 3.6 points of accuracy cost about four times as much per correct answer.
The authors write that the findings came from a "controlled benchmark setting". Microsoft's post says a prospective study with Beth Israel Deaconess Medical Center in Boston will have patients interact with the system directly.
Microsoft had given no start date for that study by 22 October. Until it reports, the 83.2% and the $1,303 are benchmark numbers from text based cases. They are not results from patients.