,
Matt Carey
When ShuffleBrain CEO Amy Jo Kim took the stage at the 2011 CasualConnect game design conference, she started with a strangely defensive proviso:
“I’m going to give you a quick run-through of how a game designer thinks about gamification,” she said, “and rather than arguing about what’s good and bad about it, I’m just going to give you an approach and some smart tips for doing a great job applying gaming principles to whatever it is that you’re working on.”
I was immediately encouraged by this introduction because I personally think the loosely connected bag of game mechanics sold under the label “gamification” is the worst thing since moldy bread, so I naturally expect any honest account of anybody’s experiences following that design philosophy to be preceded by at least a vague manifestation of guilt.
ShuffleBrain is essentially a gamification consulting agency. Companies can hire them to adorn their products with features like badges and experience levels. In the words of ShuffleBrain’s website, they can “embed game design expertise into your product team” to “develop games, services, and apps that embody the compelling pull of ‘smart games.’” Of course “gamification” is a term that makes more sense in marketing than game design, because how do you gamify something that’s already a game? I’d define it as the process of making a company’s marketing campaign more like a really awful casino game so the company’s cheap-ass customers will keep poking at the marketing materials for some interminable about of time in the hope free prizes will fall out, and then finally give up and buy something so they’ll have something to show for all their sunk time. But the message of Dr. Kim’s speech was largely about applying gamification to games themselves, which I took to mean making ordinary acceptable-looking video games function more like awful casino games.
ShuffleBrain claims to have done gamification work on the regrettable match-3 game Bejeweled 2 as well as Moshi Monsters, a vastly successful UK-based virtual pet game that now claims 60 million users. Moshi Monsters was a typical success story of the gamification gold rush, building up hapless seven-year-olds’ emotional investment in virtual pets that can only be kept happy and healthy with daily free-to-play grinding sessions, then leveraging the kids’ desperate need to feel loved into purchases of paid subscriptions and a bunch of real-world rubbish with codes on it for virtual items. These are typical guilty-pleasure types of games, the kind that don’t claim to do anything more than kill time and aren’t looking for any higher praise than to be called “addictive.” But there’s also a surprising entry on ShuffleBrain’s client list: ShuffleBrain claims to have “re-designed the onboarding process for Lumosity.com, which offers games that improve your cognitive functioning.” I had heard of Lumosity before, and to me it seemed like an unlikely partnership. Why would a company like Lumosity, which goes to so much trouble to cultivate a positive and healthy image, hire consultants who were so open about their goal to manipulate their users?
Lumosity is part of the “brain-training” genre of games, much like Brain Age on the DS or Big Brain Academy on the Wii, but unlike its competitors which prudently position themselves mainly as entertainment products, Lumosity isn’t shy about making attention-grabbing claims about its health benefits. In a , a bunch of happy young people claim to have experienced various cognitive benefits from playing Lumosity, and an announcer makes the bold claim that “No matter why you want a better brain, Lumosity.com can help. It’s like a personal trainer for your brain, using the science of neuroplasticity, but in a way that just feels like games.”
There’s also a stark difference between the way Lumosity describes its effects on players and the way Dr. Kim, drawing on her PhD in behavioral neuroscience, describes the effect of ShuffleBrain’s grind-heavy design. In “Smart Gamification” Kim talks about game design as a way of “getting people to do things.” She illustrates her point with a Powerpoint slide with a picture of a slot machine and a chart graphing “cumulative number of responses” over time with various reward schedule patterns, showing that users respond more eagerly to rewards that come at randomized intervals than they do when they can anticipate and control the outcomes of their actions.
“What happens over time is what’s known as a reinforcement schedule in behavioral psychology,” Dr. Kim says. “Patterns of reinforcement over time that have variability in them are much more addictive than very straightforward ‘do this, get that’ in a deterministic sense. Game designers know this. That red pattern which is the most addictive is exactly what’s programmed into a slot machine. And that’s what gets you addiction, surprise, and it’s actually very good at building habits. It’s that blend of something you can count on with a certain element of surprise. Those are the sorts of things that no matter what you’re doing you can build into your experience.” She also shows herself to be an avid proponent of all kinds of typical grind-design cruft, specifically endorsing points, levels, leaderboards, badges, virtual goods, appointments, and progressive unlocks.
I found Dr. Kim’s description of her design philosophy very suspicious, especially as applied to Lumosity. She seemed to be talking about finding ways to make essentially tedious tasks feel beneficial and productive—what Jonathan Blow, in a very , called “engineering around boredom.” This approach concerns me, as I tend to think of boredom as a precious evolutionary gift, a natural defense system that prevents us from whiling away our days collecting rubber bands or counting the blades of grass in our lawns. It’s an important engine for creativity, and many people willingly choose to do hard, valuable work to alleviate boredom. The way I see it, if we’re truly doing something boring, instead of allowing our natural boredom to go to waste, we should let ourselves experience it and let it direct us in more valuable directions. If gamification was the only thing keeping Lumosity from being boring, then I was very suspicious about whether Lumosity would be worth playing at all.
If people who are concerned about their cognitive health are going to expose themselves to psychological devices like the “Engagement Loops” Kim showed in her presentation to make boring video games more tolerable, they ought to have a really compelling justification for assuming that these seemingly mind-numbing activities are actually good for their brains. So I decided to actually try some Lumosity games to figure out whether their players were motivating themselves to do something healthy, like setting a schedule to remind themselves to jog or eat their vegetables, or something useless but addictive like (to take Kim’s example) playing a slot machine.
Before I begin, allow me to note that I explained my concerns and asked for comments from Lumosity, ShuffleBrain, and all five members of Lumosity’s Scientific Board whose bios indicated research interests in cognition, memory, or learning. They all either declined to comment or didn’t respond.
Feel The Burn: Training My Brain
As soon as I logged in with a Lumosity starter account in I saw the numbers “00010.” I’d been given 10 “Lumosity Points” on a scoreboard that apparently could go into the tens of thousands. Lumosity Points are one of the three scales used to measure progress for Lumosity players, the others being game scores and Brain Performance Index. Lumosity Points come from things like posting personal best game scores and playing all the games Lumosity assigns you for a play session, and as far as I can tell, no matter how badly you’re performing on the games, your Lumosity Points can never go down. In fact, because of the needlessly complex way your various scores interact in Lumosity, you can be pretty confident that no matter what you’re doing on the website at any given time, some score of yours somewhere is going up.
Much more attention goes to Brain Performance Index. Lumosity claims that “ BPI is Lumosity’s version of IQ ” and it promises in a money-back guarantee that if its two-year subscribers play Lumosity at least three times a week they’ll experience a BPI increase of at least 400. In fact, after my first session Lumosity gave me a chart showing that my BPI was projected to improve from around 450 to 850 in just the first three months of training. 450 to 850 of what , I’ll probably never find out, because beyond explaining that it’s based on a statistical distribution of game scores, Lumosity keeps the BPI formula secret as its own “ proprietary algorithm .” Nonetheless, Lumosity did helpfully announce that the increase shown on the chart would mean an 87 percent improvement in my brain (obviously using the highly scientific process of “divide the higher number by the lower number”). The message was clear: whatever the heck this BPI stuff is, it’s worth having.
But any frail illusions I had about Lumosity’s trustworthiness in reporting their game’s cognitive benefits were shattered by their blog post “ Picture Your Improvement With This BPI Chart. ” They made this claim:
On average, users who played at least 1000 games saw their BPIs more than double — the equivalent of moving from the 25th percentile to the 75th percentile. Furthermore, this holds true for people of every age and starting ability. Everyone can improve with sufficient training!
Just like an investment fund manager who says “on average my investors double their money every year” is really saying “I am running a Ponzi scheme,” a game designer who says “on average I move my customers from the 25th to the 75th percentile on an intelligence test” is really saying “I measure intelligence based on how much money people give me.” What seems obvious from these claims, which seem to indicate a heretofore unheard of increase in mental capacities over a short period of time, is that what is happening is less an increase in mental prowess and more a shifting of the goalposts, a false measure designed to suggest progress that is, in fact, fake.
Nevertheless, my first session with Lumosity was pretty pleasant. The site casts itself in the role of a personal trainer, choosing three or four simple browser games for you to play each day and chiming in with lists of mental skills you’re supposedly improving and optimistic projections of your future performance. The first game, Speed Match, involved being shown a series of shapes and pushing a button to indicate whether the current shape was the same as, or different from, the previous one. This was essentially a 1-back task, a much easier variation on the N-back task that some researchers believe can improve working memory in children. An encouraging sound played for every correct shape, as well as a Mario-mushroom power-up sound indicating a score multiplier when I got several right in a row. Lumosity said Speed Match exercised three suspiciously similar-sounding skills: “Thinking faster,” “Faster reaction time,” and “Speeding up cognitive processes.” The second game involved reproducing patterns of squares after they flashed on the screen, and the third one involved focusing on a formation of five birds where one was facing a different direction from the rest. Each right answer uncovered part of a picture of a bird. Uncover the whole bird, and I’d get to add it to my permanent Bird Journal. Fun.
I had a sense that the scores in these games might be engineered more to keep players motivated than to measure their performance. Lumosity claimed its records of player performance gave it the “largest database of human cognition ever assembled,” and that researchers were actively exploring that database “to understand the determinants of cognitive performance and cognitive enhancement,” but it seemed unlikely that Lumosity could treat game scores as scientifically valid measurements even after introducing elements like multiplier bonuses that effectively meant the same score could mean two different things for two different people.
Later I played a full game of Playing Koi , a trivially easy task involving clicking on fish, and gained a perfect score of 1050. The only problem was that 1050 was not a high enough score to get a good Brain Performance Index. So how was I supposed to increase my “performance” to meet Lumosity’s standards? The game explained that by playing once, I had unlocked a new, higher-scoring fish pond. So I played another trivially easy round, with my brain feeling closer to shutting down from boredom than it was to being “trained,” and this time received a perfect score of 1600. And then I unlocked a new pond, the per-round bonuses arbitrarily increased from 100 to 200, and so on.
The game mechanics were nothing out of the ordinary for a free Flash game, and there’s nothing more commonplace than a video game that makes you play through a few easy levels before moving on to higher-scoring ones. So why was I offended? Mainly because Lumosity claims the authority to certify that I’m now more intelligent purely because I participated in its level unlocks and point grinding. Because Lumosity calculates Brain Performance Index based on some kind of weighted average of your scores on different playthroughs, it was clear that my scores from my first few playthroughs would continue to drag down my average for as long as I play. That means that each additional time I play on the highest-scoring pond my BPI will continue to rise, even if I never improve at fish-clicking in any way , simply because I’m atoning for my unenlightened and doltish past as a person who hadn’t yet unlocked the higher-scoring levels. Familiar Faces was another game with level unlocks, requiring players to work their way up through ten levels compared to Playing Koi’s four. A look at YouTube showed that even the first Lumosity game I played, Memory Match, if I stuck with the program long enough. So if Lumosity’s plan for generating fake progress against its BPI standard wasn’t obvious from the fact that they hired ShuffleBrain for a gamification makeover, it seemed clear from a close look at Lumosity’s games. Apparently a big part of Lumosity players’ BPI improvement was simply doled out for repetitive point grinding regardless of whether they got any better at the games.
There’s a serious question about whether misleading progress reports would undermine the benefits of brain training by causing players to focus on the wrong exercises. But that question assumes the mental exercises have any cognitive benefit in the first place, which is exactly what Lumosity has been unable to prove for most of its users. In a study published in Nature in 2010 , a team of neuroscientists and psychiatrists performed a brain-training study on over 11,000 subjects, including 4,000 who were trained on tests of memory, attention, visuospatial processing and math similar to those found in commercial brain training games like Lumosity. The scientists concluded unequivocally that their results “provide no evidence for any generalized improvements in cognitive function following ‘brain training’ in a large sample of healthy adults,” and “that training-related improvements may not even generalize to other tasks that tap similar cognitive functions.” The basic problem was that Lumosity teaches to the test, much like a grade school teacher who shortchanges his students’ actual education by wasting weeks drilling them on the trivial quirks of a standardized exam. Some of Lumosity’s games might be a valid measurement of cognitive abilities for people who have never seen them before, but if you actually train on them you develop skills that are good for playing Lumosity games and nothing else.
You might wonder whether Lumosity’s fake progress could still be valuable because it motivates you to stick to your training plan and eventually reach some high-level content that’s incredibly healthy for your brain. I didn’t personally test that hypothesis, but I did find a good example of high-level Lumosity play , where a player reaches a high level in a game about solving incredibly simple arithmetic problems. Instead of making the math problems on the higher levels actually, you know, harder , Lumosity just spams the easy problems until they fill the screen. The design problem here is that players can score points and advance by giving the right answer to any of the dozens of problems on the screen, so the YouTube player maxes out his score just by mashing every number on the number pad. This player is not exploiting the game in any way—he’s playing it exactly the way it was designed. He reveals at the end of the video that Lumosity certifies him with a perfect BPI of 1700, meaning as far as Lumosity is concerned he literally has the highest-functioning brain of anybody on the planet, but it’s obvious that any notion that he’s seriously exercising his intelligence has gone out the window.
The only studies that claimed healthy adults could benefit from Lumosity were tiny and scientifically dubious—the flagship study Lumosity relies on to validate its health claims involves only 14 training subjects, and uses research methods so bogus the company couldn’t get it published anywhere other than its own website . The study trained participants on only two mental skills: visual attention and working memory. In the visual attention training, Lumosity committed the basic error of testing the subjects’ intelligence using the same test they’d just been training on (minus the cartoon graphics). In the working memory training, Lumosity’s test results simply showed no significant difference in improvement in people who played the Lumosity games versus those who didn’t
Lumosity also showcases a study it published in the Mensa Research Journal which involved only 23 participants with an average age of 54. Again Lumosity was only able to able to make its trained group barely edge out the untrained control group, and once again the test subjects only showed gains on tests that were very similar to the Lumosity games they’d been trained on. But Mensa (which has decades of experience in the business of taking people’s money to praise their supposed intelligence) nonetheless allowed Lumosity to publish their bizarre conclusion that “the benefits of cognitive training transferred to untrained measures of core cognitive abilities.” Not only that, but Mensa has now moved to cash in on the brain game business on their own. Mensa Academy has now been published by Square Enix and was released in late July on Xbox Live, PSN, WiiWare, and Steam.
To be fair, there were studies suggesting that Lumosity games had their uses for at-risk groups. Children recovering from brain damage caused by cancer and chemo seemed to experience some benefit. There was evidence that seniors at risk of dementia might benefit from brain training in general , but a study showed that Lumosity itself improved their “visual sustained attention” and not much else. As the study results trickled in after its 2007 founding, Lumosity was starting to look like, at best, a minor medical product that doctors would recommend for a limited number of patients—not at all like the nationwide phenomenon every Bay Area startup wants to become. Lumosity needed a way to appeal to a much broader audience far beyond any populations proven to experience any health benefit from their product.
It succeeded. Lumosity’s Alexa traffic rank has improved from about 6,000 to less than 2,000 in just the last two years. Lumosity now claims 20 million users, and according to Alexa, the most common age ranges for Lumosity users are over 45 and under 25 (they’re also disproportionately less affluent, more educated, female, and childless). Why would hordes of women under 25 willingly spend hours every month and pay a recurring subscription fee for a training regimen invented to stave off senile dementia? One reason might be the marketing, which presents young women as Lumosity’s star “ success stories ” and relies on their anecdotal experiences to support sweeping claims about Lumosity’s health benefits. But another likely reason is Lumosity’s barrage of misleading and outrageously optimistic feedback.
As a venture capitalist said in a 2011 interview with Fast Company , “When we first invested, we were concerned this was just a niche area for people with Alzheimer’s or other cognitive problems. But Lumosity has proved there’s universal demand for this among all demographics.”
Lumosity even partnered with the New York Times to provide the “Brain Games” feature on the Times’s website. The Times claimed that Lumosity can “improve the health and function of your brain with the right mental workouts” and that it’s “scientifically designed” by “some of the leading experts in neuroscience and cognitive psychology.” (The Times neglected to mention that one neuroscience PhD who worked on Lumosity was Amy Jo Kim, and her objective apparently was to keep players coming back to Lumosity, not to improve their cognitive health.)
The irony behind the Times making these claims was that the New York Times Magazine published an article earlier this year called “ Can You Make Yourself Smarter? ” The article quoted experts who explained that reputable studies had failed to show any transferrable intelligence benefits from “brain training.” Although the article held out hope that the newer N-back test could grant its users transferrable mental skills, most of the games the New York Times offered on its “Brain Games” page had nothing to do with the N-back test. When I emailed the Times’s PR department about how they justified the health claims packaged with their games, they refused to disclose even how long the “Brain Games” page had existed on the grounds that the Times was “no longer doing business” with Lumosity and the page should have been taken down. Indeed, within hours of my inquiry, braingames.nytimes.com began redirecting to this adorably perplexed cartoon boffin.
For the time being, the New York Times’s Brain Games page can still be seen on the “partners” section of Lumosity, and Lumosity still has many outposts at health insurance companies’ sites as well major publications like the Guardian, the Telegraph, Popular Science, and Woman’s Day. At many of those sites you can play games for free that would cost money if you went to Lumosity’s own site. Although the Times’s PR representative insisted that the Times’s relationship with Lumosity “was not an advertising relationship” and the Times’s Brain Games page was not “advertorial” content, Lumosity does market an affiliate program that allows popular websites to earn commissions on sales and leads they refer to Lumosity.
Maybe these outside sites can also help explain how Lumosity guarantees its players will progress in BPI. Nonsubscribers who encounter Lumosity games will probably sensibly quit playing after one or two repetitions and never unlock the higher-scoring versions of the same games, and even if they do unlock them they’re more likely to lose their progress because they don’t have account credentials. It would be interesting to know whether Lumosity adds nonsubscribers’ games to what it calls its “database of human cognition,” because Lumosity assigns BPI based on where a player falls on the spectrum of games that have already been played. If Lumosity is like most online games where only a small percentage of people who play for free get converted into paying customers, then it’s logical to suspect that Lumosity’s statistical curve of BPI scores is heavily weighted with players who were limited to the introductory score caps on many of the individual games. That would mean that when Lumosity subscribers picked up the score bonuses for level grinding, they wouldn’t just be told they were exceeding their old performance—they’d be told that they were outthinking the vast majority of humanity and rapidly ascending into genius territory. That could be a pretty convincing incentive to keep paying the monthly fee, at least for a certain audience.
My experience left me believing Lumosity still had questions it needed to answer for its users. When a game markets itself as a health product, doesn’t the developer have a responsibility to market the game only to users who can benefit and only to the extent that they can benefit? When players use a game for health reasons and not for entertainment, shouldn’t the developers be expected to give positive feedback only for behaviors proven to be healthy and not to engineer around the players’ natural aversion to exercises that don’t challenge them? Or should Lumosity’s health claims be considered so outlandish that no one would seriously rely on them anyway?
