Delaware study finds small AI models can better extract cancer data after fine-tuning

A Delaware research project found that compact open-weight AI models improved sharply when trained on a focused oncology task, with F1 scores rising by about 60% on extracting structured cancer data from clinical notes. The findings suggest health systems could run useful oncology tools locally, while still needing expert review and further validation.

Why it matters: - Fine-tuned compact AI models could help health systems turn free-text cancer notes into structured data without sending patient records to outside cloud services. - That setup could make oncology automation more practical for rural cancer centers and other organizations with limited staff, budget or computing resources. - Structured cancer data supports registries, clinical trial matching, quality reporting and care coordination. - Reducing manual data entry could give specialists more time for clinical judgment and may help care teams move patients toward treatment sooner.

What happened: - A yearlong Delaware research project tested whether two four-billion-parameter open-weight AI models could extract oncology data from clinical notes after task-specific fine-tuning. - The work was supported by Delaware's American Rescue Plan Act funding and run through Tech Impact's Data Innovation Lab, with Acellus Health as a subcontractor. - The project used data elements defined by mCODE, the HL7 FHIR oncology data standard. - The research focused on converting physician notes and reports into structured cancer data for registries and other downstream systems.

The details: - Fine-tuning raised Qwen3.5-4B's F1 score from 0.41 to 0.66 on a held-out test set, an improvement of about 60%. - Fine-tuning raised MedGemma 1.5-4B's F1 score from 0.22 to 0.62, nearly tripling performance. - Results were consistent across three independent training runs for each model. - F1 measures the balance between finding relevant information and avoiding incorrect extractions. - The biggest gains appeared in elements that require standardized clinical codes, which are important for registries and interoperability. - The two models have four billion parameters each, far smaller than leading general-purpose AI systems. - The findings point to task-specific training, not just larger model size, as a practical way to make clinical text computable. - Performance still varied by data element, and training did not improve every category. - Medication extraction declined for one model. - Some categories had too few test examples to support firm conclusions. - The models also differed in whether they missed documented information or supplied unsupported answers.

Between the lines: - The results suggest that smaller models may be good enough for narrow clinical workflows if they are trained on the exact task and checked carefully. - The variation across data elements shows why oncology extraction systems likely need element-by-element evaluation rather than a single overall score. - Human review remains important because the models can still omit information or hallucinate unsupported details. - Local deployment could be a strategic advantage for health systems that want to keep sensitive records inside their own environment.

What's next: - A forthcoming white paper will use interviews and project insights to explore how healthcare organizations could build practical oncology tools from this approach. - The next step is testing those applications with oncology providers to see whether they reduce data-preparation burden and care delays. - The project is preparing full methods and results for peer-reviewed publication in the coming months. - The work has not yet shown shorter treatment times or reduced workloads in clinical practice. - Any clinical use would require validation and human oversight, and the models are not intended to make medical decisions.

The bottom line: - Delaware's study suggests compact AI models can make meaningful gains on cancer record extraction when they are trained for one task, offering a path to local, lower-cost oncology tools that still need careful supervision.