Jonghyun Lee, PhD

Clinically Aligned Multi-Modal Image-Text Model for Pan-Cancer Prognosis Prediction

Jonghyun Lee, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA • Jacob S. Leiby, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA • Lina Takemaru, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; Genomics and Computational Biology Graduate Group, University of Pennsylvania, Philadelphia, PA, USA • Yidi Huang, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; Genomics and Computational Biology Graduate Group, University of Pennsylvania, Philadelphia, PA, USA • Myung-Giun Noh, Department of Pathology, Ajou University School of Medicine, Suwon, Gyeonggi-do, Republic of Korea • Jaesik Kim, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA • Byonghan Lee, Department of Artificial Intelligence, Ajou University, Suwon, Gyeonggi-do, Republic of Korea • Mattew E. Lee, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; Genomics and Computational Biology Graduate Group, University of Pennsylvania, Philadelphia, PA, USA • Derek A. Oldridge, Genomics and Computational Biology Graduate Group, University of Pennsylvania, Philadelphia, PA, USA • Young-Gyu Eun, Department of Otolaryngology-Head and Neck Surgery, Kyung Hee University Medical Center, College of Medicine, Kyung Hee University, Seoul, Republic of Korea • Hyun Jee Lee, Department of Otolaryngology-Head and Neck Surgery, Kyung Hee University Medical Center, College of Medicine, Kyung Hee University, Seoul, Republic of Korea • Young Chan Lee*, Department of Otolaryngology-Head and Neck Surgery, Kyung Hee University Hospital at Gangdong, Kyung Hee University School of Medicine, Seoul, Republic of Korea • Dokyoon Kim*, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; Genomics and Computational Biology Graduate Group, University of Pennsylvania, Philadelphia, PA, USA

Cancer prognosis prediction increasingly leverages multi-modal learning, yet existing approaches often rely on omics data that are costly and difficult to collect in routine practice. Pathology reports, by contrast, are routinely generated and widely available, providing a practical textual modality complementary to histology. We present CALM (Clinical Anchor text guided Learning for Multi-modal prognosis prediction), a general framework that integrates pathology images and reports through risk-specific anchor texts. CALM systematically incorporates prior clinical knowledge by guiding image-text alignment with large language model-derived anchors, further refined via few-shot tuning. Across 14 TCGA cancer types, CALM improved prognostic accuracy compared to image-text baselines (up to +11.5% mean C-index), with enhanced training stability. CALM achieved performance comparable to image-omics integration in PORPOISE (0.652 vs. 0.644), highlighting the prognostic value of text. External validation in an independent head and neck cancer cohort demonstrated that CALM improved generalization in zero-shot settings. Attention-based interpretation further confirmed that CALM aligns diagnostic text with tumor regions. Together, CALM offers a simple, interpretable, and clinically viable strategy for prognosis prediction, expanding the role of routine pathology reports as scalable priors in multi-modal oncology AI.