Large language models encode clinical knowledge
Karan Singhal, Shekoofeh Azizi, S. Sara Mahdavi, Hyung Won Chung, Ajay Kumar Tanwani, Heather J. Cole-Lewis, Perry W. Payne, Martin G. Seneviratne, Paul R Gamble, Christopher J. Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Dina Demner‐Fushman, Blaise Agüera y Arcas, Dale R. Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomašev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, Vivek Natarajan
Nature · 2023 · 5,232 citationsOpen access
No retraction on record in OpenAlex.Check Crossref now
Cite this paper
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A. K., Cole-Lewis, H. J., Pfohl, S., Payne, P. W., Seneviratne, M. G., Gamble, P. R., Kelly, C. J., Babiker, A., Schärli, N., Chowdhery, A., Mansfield, P., Demner‐Fushman, D., … Natarajan, V. (2023). Large language models encode clinical knowledge. Nature, 620(7972), 172–180. https://doi.org/10.1038/s41586-023-06291-2
In text: (Singhal et al., 2023)
Ask a question about this research
One question, free, no account. Anchorcite finds the papers, checks each against Crossref and links every sentence of the answer to the passage it comes from.
Abstract
Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA 3 , MedMCQA 4 , PubMedQA 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics 6 ), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today’s models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.
Related papers
- Dissecting racial bias in an algorithm used to manage the health of populations2019
- Artificial intelligence in healthcare: past, present and future2017
- Deep Learning in Medical Image Analysis2017
- Convolutional neural networks: an overview and application in radiology2018
- Chatting and cheating: Ensuring academic integrity in the era of ChatGPT2023
Metadata from OpenAlex (CC0). Citations are generated from the published record.