Healthcare video games are a subset of the burgeoning healthcare gamification market, which uses games – from board games to virtual reality games, to treat a variety of health conditions. Like the rest of healthcare in America, the healthcare gamification market is incredibly profitable and is estimated to reach a 9 billion dollar value by 2031.
In this article I will examine the recently IPO’d company, Akili Interactive Labs, and their FDA approved product, EndeavorRX. According to Akili, this video game is a supplemental treatment for children with ADHD, and claims to “target cognitive processes involved in attention function” and “engage key neural systems associated with attentional control in ADHD patients.”
I intend to show how the foundational research was flawed both in design and interpretation of results. Furthermore, I detail how the study’s invalid conclusions led to FDA approval, thus enabling Akili to profit off of families with ADHD children.
Akili Interactive Research Publications
EndeavorRx is based on five clinical studies that evaluated over 600 children with ADHD. In this article, I will focus on just one of the studies. This study, A novel digital intervention for actively reducing severity of pediatric ADHD (STARS-ADHD): a randomized control trial (referred to as the “EndeavorRx Study” for the remainder of the article), is the flagship study that is the basis for the follow up studies for EndeavorRx and the study that ultimately led to the FDA approval of EndeavorRx.
Study Description
The EndeavorRx study is a randomized, double-blind, paralleled group controlled trial of ADHD confirmed, non-medically treated children between the ages 8 and 12. For those reading without an understanding of experimental design, this study randomly splits children with ADHD into two groups, where one group is playing EndeavorRx and the other group is playing a different game. Double blind in this case means the children nor parents didn't know if the game they had was the treatment or control group, nor did the researchers know which game each individual child had. The goal of the study is to see if the children playing EndeavorRx improve on scores of attention (measured via multiple tools that can be seen in the methods section of the paper) more than the control group. Ultimately, the research is trying to answer the question of: does EndeavorRx, and the novel mechanisms that drive the user-experience of the game, significantly improve measures of attention compared to other video games.
What are the Issues with the EndeavorRx Study?
The EndeavorRx video game, the treatment group, is described as
“An investigational digital therapeutic that uses a proprietary algorithm designed to improve attention and related cognitive control processes, by training interference management at an adaptive and personalized high degree of difficulty.”
Akili states that the “game layer [is] designed to create a fun, age-appropriate experience, using built-in game rewards, missions, levels, and storytelling.” Clearly, as also seen here, the video game was built with the interests of the children in mind and was made to be as much fun as possible.
The control group is quite different. The control group’s game is similar to an enhanced crossword puzzle and is intended “to find and connect letters on a grid to spell words; points are awarded on the basis of number of words formed, word length, and the use of unusual letters.” The EndeavorRx study goes on to state that the control group game “was designed to match [EndeavorRX] on expectancy, engagement, and time on task… targeting cognitive domains not targeted by the [EndeavorRx] and not primarily associated with ADHD.” I believe there is little to no evidence to backup these claims.
The first issue with the control group is where Akili states that the control group intervention was designed to target certain cognitive domains. The exact game for the control group is never fully confirmed in the EndeavorRx study, but is implied via a footnote that the control group is a digital therapeutic used as the treatment group in this Lumosity study. The Lumosity study found some evidence that the game was able to target certain cognitive functions; however, the population in the Lumosity study was 18 to 80 year-olds. Because there is zero overlap in the age groups in the Lumosity study and the children in the EndeavorRx study, it is not reasonable to generalize the results to this younger age group. Not only is this a case of incorrectly extrapolating results, it is particularly egregious given the cognitive development differences between the two age groups.
The second issue, and I believe the most important issue with the control group, is that the design does not account for the many inherent benefits of video games for children. The EndeavorRx study actually alluded to this issue in the discussion section. The researchers state that,
“Specific mechanisms common to [EndeavorRx] and the control group may have resulted in improvements in both groups. Both interventions required continued perseverance, sometimes in the face of failure, and may have trained coping and reappraisal skills or even increased the sense of self-efficacy and mastery. Thus, any intervention that requires the patient to engage in a regular, structured setting that may include repeated failure or repetitiveness can be seen as a potential intervention for ADHD.”
This quote all but states that the researchers were aware of and unable to account for the benefits of gaming. This is backed up by the findings of the 2022 study, Association of Video Gaming With Cognitive Performance Among Children, where it was found that children who consistently played video games showed enhanced cognitive performance in areas of inhibition control and working memory over children who did not play video games. Further, there is evidence to show that the genre of the video game is important to what cognitive functions are affected. According to the article The Benefits of Playing Video Games, “playing commercially available shooter video games are comparable to the effects of formal high school and university level courses aimed at enhancing these same skills.” These are just a few examples of a rapidly growing body of research displaying the health benefits of playing video games (if interested, here, here, and here are just a few more examples).
Ultimately, these issues with the study lead to multiple very important questions being unanswered.
How do we know that the novel elements of EndeavorRx, and not the inherent benefits of video games, are the reason the game is able to target cognitive processes involved in attention function?
How do we know the differences in the control and treatment groups aren’t due to a difference in gaming genre, the game mechanics/gameplay, visual aesthetics, and/or enjoyment by the child?
How do we know if EndeavorRx is any better at neural targeting than some of the most popular video games (i.e., Fortnite, Call of Duty, etc.) played by children?
These are just a few incredibly important and also incredibly difficult elements of a study to account for. Again, from The Benefits of Playing Video Games, the authors make the point that “it is virtually impossible to choose an appropriate control condition wherein all aspects of a game (e.g., visual stimulation, arousal induction, gameplay) are kept constant across conditions and only one cognitive challenge is manipulated.” In a study where the treatment group is EndeavorRx, an action-packed game made for children, and is compared against a crossword-like word association game that has never been studied on children, there is just too much variability in the results based on the extreme differences in the two games.
How Bad Research Will Lead to Big Profits
On June 15th 2022, EndeavorRx became the first ever game-based therapeutic to be given FDA approval. This is a major decision not only for the fact it was the first video game-based therapeutic, but that it now creates a new pathway for similar interventions to get through the FDA approval process.
Because of it's FDA approval, EndeavorRx is now able to be prescribed to children by their doctor. According to Akili's Q3 2022 earnings presentation, 95% of EndeavorRx prescriptions are not covered by insurance and must be paid out of pocket. If a child with ADHD is prescribed EndeavorRx, the family can expect to pay $100 per prescription.
What is a prescription for a video game you may ask? EndeavorRx seeks to be seen as a medicine and not as a game, so rather than a family owning the game, the child gets an activation code as their prescription. Each prescription is good for five 25 minute sessions per week for four weeks. At the end of this "prescription" the child will need a "refill" which will cost another $100. If we do some basic math, if an eight year old child is prescribed EndeavorRx monthly from 8 to 12 years old, that will come out to $4,800 per child!
The company has big aspirations both in revenue and expanding to new markets. According to Akili's Q2 2022 earnings presentation, the company seeks to generate over $500 million in revenue in the next 5-7 years. They also plan to get approved for other disorders such as Covid brain fog, depression, and autism.
Final Thoughts and some Disclaimers
There are a few important points about the study, the healthcare gamification market, and the FDA I'd like to make before wrapping up this article.
While severely flawed, the EndeavorRx study has plenty of positives to it. The participants were properly randomized into treatment and control groups, they did not lose many participants to follow up, and it had enough power to make credible conclusions from its results (for those non-statistics people, power is the statistical term that states to what degree we are able to believe that the results the study is generating are valid). By all means, it's a study that is actually very close to being designed in a way that could potentially give really interesting results. Hopefully Akili or another company interested in this space can improve upon this research and create a sound study that answers the unanswered questions I outlined above.
Second, as an avid video and board gamer myself, I am a deep believer in the therapeutic use-cases of games. Since I was a kid, games have been a source of fun, stress relief, and mental stimulation. I think both kids and adults can experience many benefits from moderate amounts of gaming and I have always been skeptical of the loud opposition crowd that claims that video games are terrible for people.
Last, while I briefly went over the FDA approval of the game, I believe that this is something that needs a much deeper look into. For the sake of this article I wanted to focus on the areas I felt that I had the expertise to comment on, plus I admittedly know very little about the FDA and their approval process. But there are two things that remain very odd to me about the approval. Why did a product with such a weak evidence get through approval? And, why does the FDA not account for affordability of a product when assessing risk of a medical tool? The approval of EndeavorRx essentially states that it was approved because it showed little risk to the patient and showed at least moderate evidence of improvement of condition. While this is true in that the game did not cause much physical or emotional harm, it seems weird to me that they would not account for things like financial affordability. Families tend to not spare expenses when it comes to the health of their children and $100 per month prescriptions is a huge toll on these families, especially those with low, moderate, and middle class incomes. The bar for approval for products like EndeavorRx seems way too low and seem to make it way too easy for companies to get FDA approved and reap major profits off of minimally or unproven products.
Ultimately, it seems that Akili is taking what we already know about games and their associated cognitive benefits, and slaps an FDA approved "healthy" sticker on it so that skeptical parents will be more willing to allow their kids to play a video game. Nonetheless, I am always open to being proven wrong. I would love to hear any feedback on this article and welcome others to read the research publications on Akili's website. Thank you so much for taking the time to read this article!