Poster #19 - Quan Nguyen

PhenoGPT2: A Multimodal Fine-tuned Large Language Models for Phenotype Extraction and Normalization from Clinical Text and Facial Images.

Quan M. Nguyen, Bachelor 1,2, Umair M. Ahsan, MS 1, Zhanliang Wang, MS 1, Kai Wang, Ph.D. 1,3* 1 Raymond G. Perelman Center for Cellular and Molecular Therapeutics, Children's Hospital of Philadelphia, Philadelphia, PA, USA; 2 Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA; 3 Department of Pathology and Laboratory Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, 19104, USA

Background: The Human Phenotype Ontology (HPO) and PhenoPacket schema are increasingly used in research and clinical settings to describe rare disease phenotypes and prioritize candidate genes. However, phenotype data in Electronic Health Records (EHRs) are often unstructured, with typos, abbreviations, synonyms, negations, and variable descriptions, complicating reliable extraction and normalization. Facial images also carry valuable phenotype information, yet methods for integrating them into structured representations remain limited. Comprehensive training datasets covering the full HPO database for both text and vision modalities are virtually nonexistent, hindering model development and downstream applications such as rare disease diagnosis and gene prioritization. Large language models have shown promise in interpreting complex textual and image data to address these challenges. Methods: We developed PhenoGPT2, a multimodal LLM for extracting clinical phenotypes and demographics from texts and images, conforming to the PhenoPackets schema. The text module builds on LLaMA 3.1 8B, pretrained on the HPO database with over 18,000 HPO terms added as individual tokens, and continuously fine-tuned on synthetic and real clinical notes via model-guided pseudo-labeling approach. The vision module represents one of the first efforts to fine-tune supervised LLaVA-Med and LLaMA 3.2 Vision models on facial images annotated with phenotype labels. A BERT-based strategy pre-filters extremely noisy clinical notes in real-world records. Results: PhenoGPT2 achieved the highest F1 and normalization scores across multiple datasets compared to existing HPO extraction tools. It performed robustly on in-house clinical notes, effectively handling negations, abbreviations, typos, and synonyms. In gene prioritization, PhenoGPT2-generated phenotypes ranked causal genes in the top-10 more frequently than other methods and even outperformed human-curated labels, demonstrating superior phenotype extraction and normalization that enhances genetic diagnosis. Conclusions: By integrating multimodal data and leveraging LLMs, PhenoGPT2 significantly improves phenotype extraction, normalization, and gene prioritization, offering a promising framework for more comprehensive and standardized rare disease diagnosis.