Poster #87: Sonia Patel

AI-enabled data abstraction provides reliable results enabling clinically meaningful research

Sonia Patel, Andrew Lenart BS, MPH, Deepak Nandagopal Beng, MSc, Dr. Juliana, MD, Dr. Federica, MD, PhD, Dr. Sonika Agarwal, M.B.B.S., MD, FAAN, FCNS Dr. Ana Cristancho, MD, PhD, Dr. Riley Kessler, MD Tom, MFA, MBA

Maternal-Fetal Medicine (MFM) and Fetal Neurologist specialists lack sufficient data on long-term outcomes of patients with congenital neurological anomalies to better inform counseling. Currently, details in patients’ electronic health records are unconducive to a structured abstraction format for analysis. The Clinical Outcomes Data Archive (CODA) at the Children’s Hospital of Philadelphia utilizes an off-the-shelf large language model (LLM), GPT 5-mini, to provide a means for cost- and time-efficient data abstraction from complex clinical notes. Current language parsing tools present limited accuracy in the presence of highly technical variables and poorly organized notes. However, through specific data collection tool design and prompting, the LLM used in the CODA platform expedites data abstraction. Objective: This study aims to evaluate an LLM’s accuracy abstracting complex clinical variables from fetal ultrasound imaging reports. Methods: Pairwise agreement between 57 LLM abstracted then human validated variables across 43 anatomic structures reported in fetal ultrasound notes were performed. LLM coverage, accuracy, and accuracy for coverage (correct responses over total gold standard responses) were calculated. Results: In a cohort of n=74, accuracy ranges were [75%, 100%] and coverage ranged [72%, 100%]. Thirty-two (56.1%) and 49 (86.0%) out of 57 variables scored ≥ 90% in accuracy and accuracy for coverage respectively. The lowest performing variables in terms of accuracy were corpus callosum and upper extremity characteristics at 75% and 72% respectively. The remaining complex variables performed with an accuracy of greater than 82% and 73%. Eleven variables, including spinal anomalies, stomach, and liver, presented with 100% accuracy; three of these had 100% coverage: cerebellar echotexture, atrial septal defect, and third ventricle size. Conclusion: Overall, implications of using an LLM highlight strong potential for improving fetal specialists’ guidance for patients through generation of data previously inaccessible or limited due to the costs of manual chart review.