Health information and informatics management researchers test AI’s ability to assist in coding workflow
Artificial Intelligence is already at work behind the scenes on everything from cell phones to online shopping reviews, but does it have the capability to help make information from your medical visits – from hospital stays to well-child visits – more efficient? Researchers at West Virginia University are studying just what impact AI might play on the world of medical coding.
“Medical coding is an essential part of our health care system,” explained Megan McDougal, associate professor in the WVU Health Informatics and Information Management program. “Currently, computerized assistive coding (CAC) technology is used to assist medical coding professionals with their workflow. We decided to determine how effective AI can be in powering CAC and opening up new opportunities for coding workflow.”
McDougal, another HIIM faculty member, Ashley Simmons, and WVU Medicine colleagues Drs. Brian Dilchner, Jami Pincavitch, Ankit Sakhuja and Lukas Meadows assessed the accuracy and performance of open-source artificial intelligence (AI) large language models (LLM) including ChatGPT, Gemini, Claude and Llama in tasks critical to HIIM.
“When you visit a health care provider, you expect accurate billing for the services received and that your medical record correctly reflects your diagnoses,” Simmons said. “This study highlights the critical importance of accuracy in medical coding. A single incorrect code could negatively impact patient care continuity and the billing process. As AI continues to advance in health care, I believe technologies like large language models (LLMs) will be utilized as tools to enhance efficiency.”
During the 18-month-long research project, the researchers evaluated the LLMs ability to extract ICD-10-CM codes from patient notes. Using 50 fictional patient records, a certified coder coded the clinical notes and kept track of the time she put in and the resulting diagnoses, while the other team members fed the same clinical notes into various open-source AI LLMs while keeping track of their time and resulting diagnoses.
The goal was to compare the open-source AI LLMs’ performance with that of certified human coders.
While the purpose of the study was not to see if AI could replace humans, the question is still one they often get. But according to McDougal, the short answer is no.
“While each LLM demonstrated unique strengths and limitations, our findings consistently pointed to the same conclusion: current LLMs are not yet capable of fully replacing human coders for these tasks,” McDougal said. “LLMs still require human oversight, fine-tuning and intervention to ensure optimal accuracy and reliability.”
Next up for the research team is continuing to promote their findings and publish their manuscript.
McDougal and Simmons recently presented at the National American Medical Informatics Association (AMIA) Symposium in San Francisco. The symposium brings together thousands of biomedical and health informatics professionals from around the world to share research and leverage cutting-edge technology to improve human health.
“At the AMIA symposium, there was significant interest in our research findings and discussion on how AI is transforming medical coding,” McDougal said. “While our study shows AI's potential, it also highlights the need for human intervention to ensure accuracy. Sharing this research is crucial for advancing the future of medical coding and health informatics, helping to ensure that AI is integrated both effectively and appropriately to improve accuracy, streamline workflows and enhance patien