Theravive Home

A recently published study looked at depression detection using in-the-wild smartphone images.

“Our study, MoodCapture, explores the innovative use of smartphone technology to detect signs of depression through ‘in-the-wild’ facial expressions captured by the front-facing camera during routine phone unlocks or app use,” co-first author Subigya Nepal told us. “We aimed to assess whether these spontaneous images, which reflect authentic emotions free from the biases of self-presentation, could serve as reliable indicators for depression. Our hope was to uncover patterns and cues within these images that could predict depressive states, moving towards a method of mental health monitoring that is continuous, non-invasive, and integrated seamlessly into daily life. Such approach promises to shift traditional, often stigmatized methods of depression detection towards a more accessible, privacy-conscious, and user-friendly framework, leveraging the ubiquity and capabilities of modern smartphone technology.”

In the lab, the researchers are deeply committed to enhancing mental health care through technology. Their work consistently focuses on passive sensing technologies to assess mental health in a way that is unobtrusive and low burden for the individuals involved. Given this commitment, they embarked on their latest research project with a clear vision: to develop a continuous, non-disruptive method for detecting signs of depression. The foundation for the project was built on existing studies suggesting a correlation between facial expressions and various mental health signals. Inspired by these findings, they were optimistic about the potential for facial expression analysis to serve as a reliable indicator of depression. Their theory was that facial expressions, when captured in natural, unscripted scenarios—what is referred to as "in the wild"—could be highly indicative of depression, potentially revealing a richer array of signals compared to posed or performance-based expressions.

“We undertook this study to address critical gaps in traditional methods of monitoring and detecting depression,” Nepal told us. “Traditional approaches often involve self-reports and clinical assessments, which can be biased and may not capture the complexity of an individual's mental state continuously. MoodCapture aims to leverage the unguarded facial expressions captured during routine phone unlocks, envisioning a future where AI can assess mood in real-time directly on the device, ensuring privacy and continuous mental health monitoring. This approach signifies a shift towards more objective, unobtrusive, and continuous methods for depression detection, promising early detection and timely intervention for individuals at risk.”

To test their theory, they built on their 2015 study by enhancing the method to capture more authentic expressions. This time, they took a picture using the front-facing camera of a smartphone each time a participant answered a question from the PHQ-8 survey, a common tool for assessing depressive symptoms. Their app would take a burst of images of participants face through the front-facing camera whenever they answered one specific question. Participants consented to have their photos taken via their phone's front camera but were not aware when it was being taken. This approach allowed them to obtain more genuine and unguarded facial expressions in a seamless manner, directly linking participants' self-reported mood data with their facial expressions at the moment of reflection. This method significantly improved the authenticity of the data collected and served as a critical step in evaluating their theory with enriched detail and minimal intrusion into participants' daily routines.

“Our analysis utilized both machine learning and deep learning to evaluate MoodCapture's capability for detecting depression in real-world settings,” Nepal told us. “The findings revealed that the Random Forest (RF) model, particularly when trained with 3D landmarks, was the most effective, outperforming the deep learning models in overall classification and prediction of PHQ-8 scores. We obtained an accuracy of 61% when trying to differentiate between depressed and not-depressed states. Our preliminary experiments with personalized models obtained even better results with a boost of upto 15% in this score (meaning 75% accuracy in detecting depressed/not-depressed states). So this has potential to be a game changer in the future.”

Given the context and the innovative approach of the study, the researchers approached the results with an open mind rather than specific expectations. Initially, they viewed this study as a proof of concept, aiming to demonstrate the feasibility and potential effectiveness of using passively captured images for depression detection. The results, particularly the performance indicators such as the accuracy of 75% have been highly motivating. Although they had hoped for results that could approach a high success rate to strongly validate our concepts, the outcomes achieved thus far have significantly boosted their confidence that what they're working on is not just theoretically sound but also practically implementable.

“The significance of our findings in the MoodCapture study lies in demonstrating the feasibility of using ‘in-the-wild’ smartphone images to detect depression,” Nepal told us. “By analyzing spontaneous facial expressions and environmental factors captured in daily smartphone interactions, we provided evidence that these subtle, often overlooked cues can be meaningful indicators of mental health states. This represents a shift in mental health assessment towards more passive, continuous, and non-invasive methods. It also shows that we may be able to make depression detection more accessible and less stigmatized, by embedding it into the fabric of daily technology use. These findings pave the way for future innovations in mental health care, highlighting the potential for advanced technology and AI to contribute to early detection, timely intervention, and personalized care in mental health.”