A new study from NYU Langone Health finds that artificial intelligence produces better response rates than human providers when answering patient questions in electronic health record emails — and could help ease the burden of transcription for clinicians.
The study, led by researchers at New York University Grossman School of Medicine, found that not only did the genAI messaging tool provide accurate responses to patients’ EHR queries, but its responses also demonstrated greater empathy.
“Our results suggest that chatbots can reduce the workload of care providers by allowing for efficient and empathetic responses to patient concerns,” Dr. William Small, clinical associate professor at NYU Grossman School of Medicine and lead author of the study, said in a statement Tuesday.
WHY IT MATTERS
Doctors typically receive more than 150 In Basket messages a day during the pandemic, a more than 30% increase in daily messages compared to the same period last year, according to Dr. Paul Testa, NYU Langone’s chief medical information officer.
NYU Langone looked at how to address excessive note-taking and other documentation burdens that are major sources of physician burnout. For the new study, medical researchers asked sixteen primary care physicians to rate 344 randomly assigned pairs of AI and human responses to patient messages for accuracy, relevance, completeness, and tone.
According to researchers in the Large Language Model–Based Responses to Patients’ In-Basket Messages report, published by JAMA, the AI-generated responses outperformed human responses in terms of comprehension and tone by 9.5%.
For the blinded study, participating physicians indicated whether they used the AI responses as a first draft or had to start over. They did not know which responses were compiled by humans or the AI tool.
As a result, AI responses were 125% more likely to be perceived as empathetic and 62% more likely to use language that conveyed positivity and engagement, the researchers said.
However, they are 38% longer and 31% more likely to contain complex language.
“While humans answered the patients’ questions at a sixth-grade level, the AI wrote at an eighth-grade level, according to a standard measure of readability called the Flesch Kincaid score,” the researchers said.
They also note that future studies are needed to confirm whether personal data improves the performance of AI tools.
THE BIGGER TREND
In 2023, NYU Langone licensed GPT4 to allow physicians to test it using real patient data in a secure environment.
Also last year, the health research organization encouraged teams of clinicians, educators, and researchers to work together to test large language models in the Generative AI Prompt-A-Thon in healthcare, with no experience required.
With the event attracting 70 attendees and more than 500 more via webinar, the health system aims to use the observations gained to inform its in-house generative AI capabilities.
The researchers then used AI to look at transcripts from 820 individuals who received psychological treatment during the first wave of COVID-19 in the United States and compared them to healthcare workers.
That AI tool can detect distress in overwhelmed hospital staff — those who discuss insomnia or mood issues in therapy sessions are more likely to be diagnosed with anxiety and depression.
ON PROFILE
“We found that EHR-integrated AI chatbots using patient-specific data can compose messages of similar quality to human providers,” Small said in a statement.
“With physician approval, GenAI messages will in the near future be on par with human-generated responses in quality, communication style, and usability,” said Dr. Devin Mann, senior director of informatics innovation at the Center for Information Technology at NYU Langone Medical Center.
Andrea Fox is senior editor of Healthcare IT News.
Email: [email protected]
Healthcare IT News is a publication of HIMSS Media.
