Illustration of a computer with electronic health records on it that can be managed using AI-generated chatbot messages
Credit: GrafVishenka / Getty Images

Research led by New York University shows an artificial intelligence (AI) driven chatbot can answer patient queries about their medical records in an accurate and empathetic manner.

During the COVID-19 pandemic, digital communication between patients and physicians increased enormously. One example is via electronic health records. Patients have access to their electronic health records and are able to ask questions via an email service.

Clinicians, particularly primary care physicians, are now receiving hundreds of these messages a day and simply do not have time in their working day to answer all of them. Indeed, the burden of having to answer so many messages has been linked to physician burnout.

William Small, a clinician and assistant professor at NYU Grossman School of Medicine, and colleagues carried out a study to test the quality of generative AI-produced answers to patient queries and assess how well they would be received.

Reporting in JAMA Network Open, Small and team found that AI-generated messages were of similar quality to those written by physicians and were even scored higher for empathy and communication style than those written by physicians.

The study involved 16 primary care physicians (50% female) reviewing 344 messages, 175 of which were AI generated and 169 of which were physician generated, in a blinded fashion.

The AI messages received an average score of 3.7 and 3.53 for communication style and information content, respectively, versus 3.38 and 3.41 for physician generated messages.

The physicians scoring the messages rated 37% of the AI messages as empathetic versus 16.5% of the physician messages, something the researchers believe may be due to use of more subjective and positive language.

Although they were generally well received, the AI-generated messages were longer and more linguistically complex than the physician derived messages, which the authors say may be “a significant concern for patients with low health or English literacy.”

“This work demonstrates that the AI tool can build high-quality draft responses to patient requests,” said lead author Devin Mann, senior director of informatics innovation in NYU Langone Medical Center Information Technology.

“With this physician approval in place, GenAI message quality will be equal in the near future in quality, communication style, and usability, to responses generated by humans.”

Also of Interest