Theravive Home

A new study published in NEJM AI looked at a generative AI chatbot for mental health treatment.

“We conducted the first randomized controlled trial to evaluate a therapy chatbot driven entirely by generative AI,” study author Nicholas Jacobson told us. Jacobson is an associate professor of biomedical data science and psychiatry at Dartmouth. “We developed the bot over 5.5 years and the goal of the work was to treat depression, anxiety, and eating disorders and all of their related comorbidities.” 

The study involved a team of over 100 people contributing over 100,000 human hours in developing Therabot. The goal was to determine whether they could meaningfully impact and reduce depression, anxiety, and eating disorders symptoms. They also aimed to understand the degree of the working relationship between the participants and Therabot, a construct similar to what is among the best evidence-based constructs that forms the bedrock of how psychotherapy works. They also wanted to study the level of engagement with this tool given that it can produce personalized dynamic responses.

“Having designed and tested the system for 5.5 years, we believed that the symptom would produce large benefits in each of the targeted mental health symptom areas (depression, anxiety, eating disorders),” Jacobson told us. “Our work focuses on developing personalized scalable treatments. We chose this area because we seek to have real world impacts and there is a massive shortage of providers -- approximately 1,600 patients with just an anxiety and depressive disorder per mental health provider of any kind per year. The system is nowhere near ready to scale to those in need and as such most persons with a mental health disorder don't receive any treatment for it.”

The research team conducted a national randomized controlled trial of 210 adults who screened positive for major depressive disorder, generalized anxiety disorder, or clinically high risk for feeding and eating disorders. Participants were randomized to either receive Therabot or a waitlist control group during the eight-week study period. The intervention group interacted with Therabot via a smartphone application. 

“The results suggested significant, large, and clinically meaningful reductions in depression, anxiety, and eating disorder symptoms at both the four-week and the eight-week outcomes,” Jacobson told us. “Users engaged with Therabot for over six hours on average and reported a strong therapeutic alliance, comparable to those seen between patients and human therapist in outpatient settings. User satisfaction was also high.”

The research team expected that they would have impacts on depression, anxiety, and eating disorders symptoms, but the degree of the change was pretty “astounding”, Jacobson told us.

“It was really what would be expected from a gold-standard dose of outpatient cognitive-behavioral therapy by expert providers,” Jacobson told us. “Additionally, it was pretty striking in the level of the therapeutic bond between therapist and the users. We didn't quite expect that the strength of the therapeutic relationship would match the strength of the bond with human providers.”

The researchers believe their results suggest that a carefully developed generative AI can produce strong, safe, and effective treatments for depression, anxiety, and eating disorders. However, they impart the need to be cautious as this isn't suggesting that all generative AI mental health products are safe and effective and by and large, most of the products on the market haven't been designed in careful or rigorous ways.

“Many are starting into the space trying to just put together a prompt with an API call to one of the large foundation models,” Jacobson told us. “Generative AI frequently produces harmful responses when it isn't actually very carefully designed so please proceed with these untested products with caution. Nevertheless, the rigorously designed systems will continue to only get better, so I'm optimistic that this could have a meaningful impact on the ability to drive strong clinical outcomes.”