Eye-tracking data trains an OCT reading assistant
A UIST paper presents Co-Annotator: expert gaze and dictation train two aids for OCT reading. Only one source backs the figures so far.
Illustration: a darkened reading room where warm color patches over layered cross-section images suggest where a viewer's gaze lingers.
Co-Annotator turns recorded eye movements and dictation from experienced retina specialists into two assistance features that, according to the paper, raised correct diagnoses per minute by 40 percent and cut comment editing time by 67 percent.
At a glance
- Preprint posted August 31, 2026, 23 pages, 11 figures; accepted for UIST '26 in Detroit, November 2-5, 2026.
- Two components: a gaze-aligned Vision Transformer for areas of interest, and an ontology-bounded VLM for biomarker summaries.
- Three user studies: US1 gathers expert data, US2 tests each aid alone, US3 combines them at two academic institutions.
- Figures from US3: 40 percent more correct diagnoses per minute, 67 percent less time editing comments.
- Not stated: participant counts, model sizes, a code link, or a comparison against other assistance systems.
Co-Annotator turns recorded eye movements and dictation from experienced retina specialists into two assistance features that, according to the paper, raised correct diagnoses per minute by 40 percent and cut comment editing time by 67 percent. The team around Ziheng Li and Kaveri A. Thakoor posted the preprint to arXiv on August 31, 2026. The work is set to appear at UIST '26 in Detroit, November 2-5, 2026.
The target is the workflow, not the accuracy score
Clinical AI tends to chase predictive performance while ignoring how clinicians decide where to look and what to write — that is the opening claim of the abstract. Co-Annotator therefore aims at two bottlenecks in the daily routine instead: the visual search across the image, and the paperwork that follows it.
The setting is optical coherence tomography of the retina, the workhorse imaging method in age-related macular degeneration. One component is a gaze-aligned Vision Transformer that derives areas of interest from where specialists actually fixate. The other is an ontology-bounded vision-language model that pre-fills a biomarker summary as an editable draft rather than a finished report.
Three studies, three different jobs
The first study existed to collect the expert gaze and dictation the models learn from. The second, a controlled study with residents, isolated the two aids: the area-of-interest overlay improved perceptual efficiency, while the language model more than doubled the breadth of biomarkers that got documented, per the authors.
Only the third study combined both in deployment across two academic institutions, and that is where the 40 percent and 67 percent figures come from. The authors state that diagnostic accuracy did not suffer, and frame the result as clearing two separate bottlenecks at the same time.
What this report cannot confirm
Exactly one independent source covers this work so far: the arXiv page for the preprint. No second newsroom has checked the findings, and AI IN LIFE opened no other source. Every number above is the authors' own reporting.
The abstract and metadata page gives neither participant counts nor model sizes. It also omits the names of the two institutions, a link to source code, and any comparison against existing assistance tools. Whether the percentages hold beyond those two sites is therefore an open question that this source cannot settle.
Why it is still worth tracking
Using gaze as a training signal in image reading is not a new idea; pairing it with a pre-filled, ontology-constrained text draft is what this paper actually proposes. Acceptance at a user-interface conference places the project in human-computer interaction, and it is cross-listed under artificial intelligence. The picture only firms up once independent groups reproduce the numbers on their own patient data.
FAQ
What does Co-Annotator actually do?
It highlights regions of an OCT scan that experienced readers typically fixate on, and it pre-fills an editable draft of the biomarker summary. Both aids are learned from recorded gaze and dictation supplied by specialists.
How solid are the 40 percent and 67 percent figures?
They come from the authors' own third user study, published as a preprint. We have no independent replication, and the abstract page lists no participant counts that would let readers judge the sample.
Is the system already used in clinics?
The source describes combined use at two academic institutions as part of the third study. It says nothing about regulatory clearance, a commercial product, or routine use beyond that study.
