A new study from Mount Sinai shows that using artificial general intelligence to support encryption automation has some significant limitations.
WHY IS IT IMPORTANT?
In the study, Mount Sinai’s Icahn School of Medicine evaluated the potential application of large language models in healthcare to automate the assignment of medical codes – based on clinical text – for the purpose of reimbursement and research.
The study compared LLM from OpenAI, Google, and Meta to evaluate whether they could effectively match medical codes to the corresponding official text description.
To evaluate and compare the performance of GPT-3.5, GPT-4, Gemini Pro, and Llama2-70b, researchers extracted more than 27,000 unique diagnosis and procedure codes from 12 months of routine care within the Mount Sinai Health System, does not include patient data.
“The Previous research indicates that newer large language models have difficulty with numerical tasks.” .
“However, their accuracy in assigning medical codes from clinical text has not been thoroughly studied across different models.”
In evaluating whether four existing models could effectively match medical codes through qualitative and quantitative methods, the researchers identified all LLMs achieving less than 50% accuracy. in generating unique diagnostic and procedure codes.
While GPT-4 performed best in the study with the highest exact match rates for ICD-9-CM at 45.9%, ICD-10-CM at 33.9% and CPT codes at 49 ,8%, there is still an “unacceptably large” error.
Researchers said GPT-4 produced the most incorrectly generated code, while GPT-3.5 tended to be the most ambiguous, identifying generic codes rather than correct codes.
Research results New England Journal of Medicine AI announced last week, leading researchers to warn that the LLM’s performance in real-world medical coding could yield worse results.
“LLM is not suitable for use in medical coding tasks without additional research,” the researchers said in the report.
Dr. Ali Soroush, assistant professor of D3M and medicine, warned in a statement: “While AI has enormous potential, it must be approached with caution and continuously developed to ensure accuracy. Reliable and effective in health care.
Mount Sinai noted that researchers will look to develop appropriate LLM tools to extract accurate medical data and assign billing codes.
BIGGER TREND
Despite the Mount Sinai study’s findings, others see value in AI-enabled coding and say AI systems can help physician groups avoid missed revenue opportunities and improve compliance their documents.
Dr. Bruce Cohen, surgeon and former CEO at OrthoCarolina in Charlotte, North Carolina.
Dr. Bruce Cohen, a surgeon and former CEO at OrthoCarolina in Charlotte, North Carolina, told Healthcare IT News: “Once annual coding requirements are established, an AI-based system incorporate and implement those changes in real time.”
AI-based systems don’t eliminate the work of programmers, he added: “It expands the visibility and accuracy of any charges incurred based on management review and coding.”
ON PROFILE
“Our findings highlight the critical need for rigorous assessment and screening before deploying AI technologies in sensitive operational areas such as medical coding”.
Dr. Girish Nadkarni, director of the Charles Bronfman Institute for Personalized Medicine and systems director of AI, added: “This research sheds light on the current possibilities and challenges of AI in healthcare, emphasizes the need for careful consideration and additional refinement before widespread adoption.” D3M.
Andrea Fox is a senior editor at Healthcare IT News.
Email: [email protected]
Healthcare IT News is a publication of HIMSS Media.
