Poster #99 - Jaesik Kim

Evaluation of Multi-Agent LLMs in Multidisciplinary Team Decision-Making for Challenging Cancer Cases

Jaesik Kim1, Byounghan Lee2, Kyung-Ah Sohn2, DokyoonKim3, Young Chan Lee4 1Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA 2Department of Artificial Intelligence, Ajou University, Suwon, Republic of Korea 3Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA 4Department of Pathology, Kyung Hee University of Hospital at Gangdong, Seoul, Republic of Korea

This study explores the potential of large language model (LLM) agents in real-world clinical decision-making, focusing on their alignment with human experts in cancer multidisciplinary team (MDT) meetings. While LLMs perform well on benchmark medical questionanswering tasks, these evaluations often oversimplify the open-ended, multifaceted nature of actual clinical decisions. In practice, MDTs require balancing diverse expert opinions and multiple valid treatment options. Using real MDT meeting data, we compare different LLM approaches including single-agent and multi-agent systems to assess their ability to replicate consensus-based decisions. Our findings indicate that multi-agent, conversation-based systems, which assign specialized roles and facilitate dynamic inter-agent conversation, better align with human expert decisions in our data. Overall, this work highlights the potential practical utility of LLM agents in complex clinical settings and lays the groundwork for their future integration as decision support tools in multidisciplinary medical contexts.