James Liu, Ryan Kua, A/Prof. Khoa N. Cao

Few benchmarks are reflective of real-world clinical medicine. This project aims to develop LLM benchmarks which are reflective of how medical doctors operate in the real world. Our first benchmark, the Monash Murtagh Benchmark, examined the performance of LLMs in the diagnosis of over 2,500 case studies based on John Murtagh’s General Practice.

Bryan Lee, Deepak Rajan, A/Prof. Khoa N. Cao

This project aims to build a full-stack medical AGI agent for deployment in the real-world.

Lucy Liu, Izabella Mancewicz, A/Prof. Khoa N. Cao

Few studies have examined the desirable characteristics of medical LLMs. This project aims to comprehend and assess positive and negative behaviours expected of medical doctors in medical LLMs, commencing with the economic restraint or judiciousness of medical AI (i.e. AI’s ability to choose investigations and treatments wisely).

Izabella Mancewicz, Dr. Yufei Xu, Dr. Jeffrey Ma, A/Prof. Khoa N. Cao

The extent and nature of LLM use by different medical specialties remain poorly understood. This project aimed to understand how medical doctors across specialties are utilising commercial LLMs through analysis of the Anthropic Economic Index, a dataset which captured millions of conversations in Anthropic’s Claude product.

Chun Joo Goh, James Liu, A/Prof. Khoa N. Cao

Cybersecurity considerations for medical LLMs are not well comprehended. This project aims to assess the cybersecurity risks and mitigation strategies before and following LLM deployment to inform information technology systems and hospitals looking to deploy LLMs.

Bryan Lee, Chun Joo Goh, Deepak Rajan, Nethum Devendra, A/Prof. Khoa N. Cao

Significant effort is required for contribution to clinical registries. This project examined, with support from Alfred Health, how to build an AI system which automatically summarises quality management metrics and clinical registry submissions from EMR data.

Bryan Lee, Chun Joo Goh, Deepak Rajan, A/Prof. Khoa N. Cao

Guideline under-utilisation is a widespread problem across hospitals and clinics, due partly to difficulty navigating vast amounts of recommendations and literature. This project is a collaboration with Monash Health to build a retrieval-augmented generation system which provides hospital staff with AI-assisted access to both clinical and non-clinical guidelines.