The News, Explained
Microsoft AI announced MAI-DxO research results on October 22, 2026. The system coordinates virtual diagnostic roles to choose the next question or test. Its evaluation examines both diagnostic accuracy and the information requested along the way. Source
Read accuracy alongside the conditions
SDBench is a framework for evaluating diagnostic processes. It includes 304 complex cases published in the medical journal NEJM. Microsoft reports the following results for that case group, separating physicians’ access to reference materials. Source
| Evaluated system or physician condition | Reported diagnostic accuracy |
|---|---|
| MAI-DxO | 83% |
| Physicians without external references | 22% |
| Physicians using textbooks and the internet | 36% |
An evaluation starts with a short case summary. Participants request more information before submitting a diagnosis, receiving evidence in a different sequence from a test that supplies every result upfront. Readers can therefore examine the route to the answer as well as the answer itself. Source
Who supplies information and scores the answer?
The evaluation separates three roles. The diagnostic agent chooses whether to ask a question, request a test or commit a diagnosis. A gatekeeper supplies information: it returns a result from the case text when available and generates a synthetic result when missing. That synthetic information lets the evaluation continue. Source
A judge compares the submitted diagnosis with the reference diagnosis. The README specifies a score from one to five, counting four or five as correct. The chosen configuration fixes gatekeeper and judge models and instructions when comparing agents. The information provider and scoring rules belong alongside the accuracy figure. Source
How are test costs estimated?
The README says costs reported in the paper were grounded in a public hospital price list for tests and procedures. The tool also supports estimates made by a language model alone, but warns that the two methods are not directly comparable. Interpreting a saving requires matching the requested tests and costing method; treating it as a measurement of patients’ actual bills would exceed the evaluation’s scope. Source
Code and case data have separate access conditions. NEJM cases require licensing. MIMIC hospital-record data require credentialing, training and a data-use agreement. The five bundled synthetic cases are for development and testing, and are unsuitable for reporting diagnostic performance. Source
Microsoft plans a prospective evaluation with Beth Israel Deaconess Medical Center, with patients interacting directly with the system. A prospective evaluation collects new information after the study is planned. The announcement describes comparison with physicians’ independent assessments as future work. Source
OYOPICK’s Take
The useful shift is asking an AI to choose what information to obtain next. Evaluating those requests alongside test costs can reveal whether higher accuracy comes with excessive testing. That gives readers a more concrete basis for assessing medical AI progress.
For an open evaluation framework to support reproduction and scrutiny, data-access conditions and scoring settings need to travel with it. We hope other researchers can examine how synthetic information and cost estimates affect results. The planned hospital evaluation will need to establish benefits and burdens in real workflows and for patients before these expectations rest on firmer evidence.