Is the Science Behind Mental Health Apps Sketchy?

By Sabrina Rojas Weiss
November 18, 2015

They promise to make you less anxious. They promise to make you focus on the positive. They promise to help you meditate and make you happier. The claims from mood-improving apps out there are difficult to resist. Play a game and we’ll feel better about life? Yes, please!

Is it all too good to be true? Psychotherapist and former statistician Irene Gorodyansky takes the entire mental health app sector to task onTechCrunch last week, writing that the stats they tout to back up their software are misleading. Some employ studies done (or at least co-authored) by their own scientists, which means the experimenters are biased toward a positive outcome for their product.

“I was disturbed when I saw mental health trivialized through marketing ploys with unfounded claims of success,” Gorodyansky told Yahoo Health in an email. “These apps create false hope that mental health can be achieved through a quick fix, rather than by taking the time to understand and work through problems.”

In the case of Joyable, Happify, and SuperBetter, she says, the companies advertise really high success percentages taken from a sample group that only includes people who used the app for a prolonged period of time, with what’s called a “survival bias,” meaning the ones who didn’t like it or didn’t find it effective weren’t counted. Some of the experiments, such as one testing Personal Zen, used a small sample size, with the participants coming from a self-selecting population (college psych students, self-help seekers), and the control group couldn’t be completely blind to their status if they were not playing any games at all.

The scientists behind these apps don’t think Gorodyansky’s assessment is tells the whole story. For one thing, she doesn’t take into account the fact that apps like Happify have based all of their games on peer-reviewed scientific literature about positive psychology and neuroplasticity. When they first log in to the site, users complete a short quiz and then are given different recommended “tracks,” such as “Conquer Negative Thoughts” or “Cope Better With Stress,” and then they’re encouraged to play games to achieve these goals. On each game there’s a “Why It Works” link that takes users to an explanation of the science behind it, citing the researchon which each game is based.

“We came from a foundation of starting with things that had been scientifically validated,” Acacia Parks, Happify’s chief scientist and an associate professor of psychology at Hiram College, told Yahoo Health. “Which doesn’t excuse us from scientifically validating our product as well, and that’s something that’s well under way. But it makes us much more comfortable with the product that we’re offering.”

In her story, Gorodyansky pokes holes in a stat that appears on Happify’s site, which says “86 percent of frequent users get happier.” Because of survivor bias, she writes, that’s like “asking people at a Nickelback concert whether they like Nickelback.”

Parks says that number comes from a study she did to test how frequently users needed to play Happify games for it to be effective, and her conclusion was that moderate to frequent users who played games at least one or two times a week saw the most results.

“When people do not get better, they may feel somehow flawed because the app did not work for them, but worked for so many others, according to the marketing claims,” Gorodyansky told Yahoo Health.

“We’re specifically careful to say we are not a therapy, nor do we suggest that we provide medical treatment,” Happify president and co-founder Ofer Leidner points out, and the site doesn’t offer its services as a quick fix but rather a long-term habit. The way the app checks in with users, measuring their happiness periodically through questionnaires, helps them judge for themselves how well it’s working for them.

These apps exist more in the realm of self-help books and diet advice than professional medical help, its creators argue, though they’re still very devoted to the idea of scientifically proving their worth.

“I’m really thrilled that she’s bringing science to the forefront of this digital mental health revolution; that needs to be there,” Tracy Dennis-Tiwary, a professor at Hunter College in New York and the creator of thePersonal Zen app, told Yahoo Health. She developed the app, which asks users to focus on a happy face on the screen and ignore a mean one, as part of her research — not the other way around.

“There’s no money involved and no conflict of interest [in my research], but in the future there could be,” Dennis-Tiwary said. But even if there were a profit motive, she compared the process to the way pharmaceutical companies run their own clinical trials first and allow others to observe that they followed proper scientific procedures like double-blind control groups. Outside scrutiny should be there too, she agrees.

“There’s a mental health revolution going on before our eyes, and we have no manifesto, no guidelines, no standards,” Dennis-Tiwary said. “We need to have standards that consumers can access and understand. Right now it is the Wild West. I think science is the way to find the signal in the noise. It’s also important not to throw out studies because they’re imperfect, because science is imperfect.”

So, how can consumers without psychology degrees judge whether or not to use an app to improve their mental state?

Gorodyansky thinks companies’ claims about mental health should be subject to regulations, the way medication is. Actually, Happify’s Parks and Leidner agree — if the software is making medical claims, which theirs technically isn’t. Instead, it’s up to us to judge how well they work, the way we would for advice on how to maintain our physical health.

Originally published by Yahoo Health