Two Researchers Challenged a Scientific Study About Violent Video Games—and Took a Hit for Being Right

This story is a product of a collaboration between Motherboard and Retraction Watch.

In 2012, a prominent voice in the debate over the potential dangers of shooter games published a study titled “Boom, Headshot!” which suggested that playing violent video games helps people become better sharpshooters in real life. It was a controversial finding, and one that didn’t sit right with everyone.

The paper was co-authored by Jodi Whitaker and Brad Bushman, a prominent researcher at The Ohio State University (OSU) who often speaks to the media about the potential harms of video games, including their possible role in triggering violent behavior such as school shootings. As soon as outside experts Malte Elson and Patrick Markey read “Boom, Headshot!,” they knew something was wrong. Did the paper truly illustrate, as it claimed, “the powerful potential of video games to teach or increase skills, including potentially lethal weapon use?” Markey—a researcher who has argued against real-life impacts from violent video games—noticed some statistical irregularities. When he got the raw data, he discovered other issues that, once addressed, significantly weakened the conclusions of Bushman and his co-authors.

And they were right. After Elson and Markey raised their concerns with the authors, journal, and the institution, in 2015, OSU determined the study was problematic, and initiated a misconduct probe into the first author, Whitaker.

Then, in December 2016, the tables turned on Elson and Markey: They learned that they were under investigation for publishing confidential materials from OSU’s investigation as part of a campaign to raise awareness about the flawed study.

At that time, OSU filed a complaint against Elson and Markey, arguing they had violated OSU’s confidentiality provisions by creating a website and posting the materials, which then got picked up by Retraction Watch. Elson and Markey’s institutions—Ruhr University Bochum (RUB) in Germany and Villanova University in Pennsylvania, respectively—initiated probes looking into OSU’s concerns. Villanova cleared Markey within weeks; Elson, however, remained under investigation for one year. During that time, “Boom, Headshot!” was retracted, and Whitaker lost her PhD.

The researchers say they released the documents out of frustration with how long the investigation was taking, given that the paper was being cited in such a high-stakes area of research concerning public policy. Ohio State University claimed that the confidentiality provisions are necessary to complete its investigation process, which, in this case, “was not over.”

In response to Elson and Markey’s decision to publish confidential documents, OSU has become even less transparent during investigations into academic misconduct. The university used to share nearly all materials included in an investigation—such as correspondence, interviews, and analyses—with the people whose allegations prompted the probe as a courtesy, in order to include them in the investigation process. By being included, complainants could monitor whether their concerns were being taken seriously, and that the investigation was being conducted fairly. Now, OSU plans to scale back what it will share with complainants; anyone who makes allegations in the future could be somewhat shut out.

Scientists always face incredible risks when they take concerns about research public, and any additional pushback makes the process even harder to ensure that justice is being done. If you’re going to be punished for pointing out bad science, why bother?

After “Boom, Headshot!” appeared online in 2012, Markey noticed what he called in an interview for this article a “weird statistical anomaly.” In September 2013, he contacted Bushman to learn more about it, and asked for the original data so Markey and his team could analyze it themselves. According to Markey, Bushman initially said Whitaker would provide it, then changed his mind a few days later. “That made me suspicious,” Markey told me.

When I contacted Bushman, an OSU spokesperson and the university’s research integrity officer told me that they would speak on his behalf: “Dr. Bushman did send Dr. Markey the data set he had for the study. He simply wanted to examine the data himself first,” the spokesperson told me in an email.

In April 2014, Markey emailed the journal that published the paper, Communication Research, to intercede. One week later, he got the data, although it was edited after the paper was published and missing a variable. Still, it was enough for Markey to take a look, and he noticed some strange elements.

After study participants played video games that were more or less violent (Resident Evil 4 versus Super Mario Galaxy), they were given a training pistol with rubber bullets and placed in front of a six-foot-tall mannequin, located 20 feet away—close enough that people could aim for whichever body part they preferred. The paper reported that people who played a violent shooting games using a controller shaped like a gun fired more shots to the mannequin’s head. But some users had an unusually large number of headshots, and it appeared as if the researchers had made some errors in assigning players to the wrong experimental conditions. Of course, mistakes happen, but Markey did some calculations, and determined that the odds of these mistakes happening by chance were very small.

After some back and forth with the authors of “Boom, Headshot!” about the errors, one of the editors of Communication Research told Markey in June 2014 that he had been too busy to look into the issue (Markey forwarded us the emails.) The paper was published in print in September 2014. None of the data had been changed.

Around the time the paper appeared in print, Markey reached out to some colleagues for what he called a "sanity check" on his concerns about the data and some help pursuing them, as he was going through a divorce and couldn’t keep at it on his own. Elson—another critical reader of literature about violent video games, now the head of the Psychology of Human Technology Interaction lab at RUB—responded, and agreed that the errors Markey identified couldn’t be just minor inaccuracies. They decided to write a report to OSU of their concerns. On January 23, 2015, Markey and Elson—who told me they don’t take any money from entities that sell or profit off video games—submitted a report to OSU, which began an inquiry.

In September 2015, Elson arrived to his desk to find a flash drive, sent directly from OSU. It included a trove of information: documents and data files that had been sequestered from computers, files with track changes so they could see who changed what and when, along with personal emails between the authors, some of which included their phone numbers. Elson and Markey never asked for the information, and OSU never asked them to sign any confidentiality agreement before sending it.

“That was very puzzling to us,” Elson told me over the phone. “It was puzzling to us that they would send us all of these files that we never asked for.”

What they saw was very interesting. Among the sequestered materials, Markey and Elson found two different datasets. One had been used in the study, but there was an entirely different dataset, as well—which contained some key changes. By comparing the two datasets, it was clear to Markey and Elson that some individual players had been recategorized. For instance, in one dataset, one player was coded as having played a nonviolent game using a “pistol-shaped controller” (the study did not further specify, but controllers shaped like guns do exist), but in the other data set, the same participant was coded as having played the violent video game with this controller. The changes seemed to help support the authors’ hypothesis that people take more headshots after playing violent video games, Markey told me.

“We don’t know who—but it appears that somebody altered the datasets,” he said.

After sending the files, OSU repeatedly reminded Markey and Elson in email correspondence that university procedures required them to maintain confidentiality. “But we never agreed, either in writing or otherwise, not to talk about this publicly,” Elson said. “We always maintained we would respect their procedures at Ohio State, but we didn’t forfeit our ability to talk about it or use it for anything.”

An OSU panel concluded on October 30, 2015 that there was sufficient concern to warrant an investigation into Whitaker, but not Bushman, and that the paper should be retracted. Bushman asked the journal if he could attempt to replicate the data, which he believed were valid, before taking the final step of retracting it altogether. The journal agreed.

So Markey and Elson waited. Every so often, they would ask OSU or the journal for an update, but none came.

At this point, Markey and Elson felt like they were placed in a difficult position. Other researchers were citing the paper; it was included in a 2015 technical report by the American Psychological Association about research into violent video games.

“We knew the paper should not be in the literature,” Elson said. “At one point we felt that we became complicit in a cover-up situation, where clearly there are parties interested in not making this fact public. And we had known at this point for two years that this paper shouldn’t be taken seriously.”

They discussed the possibility of making it all public—posting the confidential documents from OSU’s investigation on a website. “We spoke about it at length,” said Elson. “Because we knew that this would not make Ohio State very happy, and of course we knew it wouldn’t make the authors very happy. But there were plenty of opportunities for them to do the right thing.”

In the fall of 2016, Markey and Elson published a website with a detailed timeline of correspondence and and uploads of the materials OSU had provided to them after their initial complaint.

On December 27, 2016, RUB received a complaint from The Ohio State University, noting that Elson had violated the institution’s provisions by releasing confidential documents before the university had completed its investigation process. Within one month, Markey (whose institution had received the same complaint) was cleared; Elson, however, found himself in a meeting with the RUB ombudsman.

Elson said he was “devastated” from the news that he was the subject of a complaint. “It happened right after Christmas,” he told me. “I’m not a religious person, but of course I wasn’t expecting anything like that at that time. Universities are quiet at that time. I was surprised.”

He was also nervous. Although labor laws in Germany tend to protect employees, Elson was a postdoctoral fellow with no guarantee about his future. “I was concerned because, even though I was sure I couldn’t be fired for it, I don’t have tenure, so this isn’t great to have on your record. And the people involved at RUB with regards to this complaint might be involved in future decisions about tenure-track jobs. This wasn’t a great situation for me.”

The meeting with the ombudsman was professional. “We spoke for two hours, maybe, and there was a lawyer present,” Elson said. “That was all a bit weird for me.” The lawyer wasn’t Elson’s, and he was unclear whose interest the lawyer represented. They talked about “everything,” said Elson. “Everything, from the start. He wanted to have a full account of all the events.” The lawyer didn’t say much.

It’s unclear why Villanova cleared Markey so quickly, while RUB took one year. When I asked RUB, I received a response from Ulf Eysel, RUB’s Ombudsman for Good Scientific Practice, who confirmed Elson was initially asked to respond to OSU’s complaint in writing and during an in-person interview. “Subsequently we scrutinized this complex and long-lasting issue (2014-2017) in detail with respect to the rules of good scientific practice,” Eysel wrote me in an email in May.

“I think I emailed [the ombudsman] every month asking about the status of this,” Elson said. During that time, the complaint was “on the back of my mind constantly,” he added. “I was extremely careful about private or public statements about the research by the authors of that paper, who happen to publish in the same area as I am.” He took down the website shortly after OSU filed its complaint, and declined to help other people when they approached him for advice about their suspicions related to other research projects. “I laid low,” he explained.

When I reached out to Bushman, I got a response from OSU’s research integrity officer, Jennifer Yucel, and Jeff Grabmeier, a spokesperson for OSU, who told me the university didn’t make Elson and Markey sign a confidentiality agreement before sharing confidential documents with them. “We didn’t think we had to—we treated them as colleagues,” Grabmeier told me during a phone call in May. “We just thought they would be as interested as we are in protecting the confidentiality of these documents.”

By violating that trust, Grabmeier said, Elson and Markey interfered with OSU’s investigation process, which is designed to protect both individual researchers and the integrity of the research. “Our process was not over,” said Grabmeier.

“Anytime you put stuff out there that’s not supposed to be out there, it hurts our ability to protect our researchers,” he said. “[The investigation process] may seem like it takes a long time, but you want to make the right decision in the end.” As a result of Markey and Elson’s decision, OSU has since changed its policy. Before, it shared documents with people who made allegations of academic misconduct, but that’s changing. “Exactly how much the information we provide to complainants will be scaled back has not been determined yet,” said Grabmeier. “It’s a shame,” Grabmeier said.

Others have questioned Bushman’s work without incident—since 2016, Joseph Hilgard has approached Bushman and his colleagues regarding two papers about the real-life impacts of violent video games. The authors shared the data, and the journals took action. One journal flagged a 2017 paper showing the presence of weapons increased aggressive thoughts with a note warning readers of errors, another retracted a 2016 paper that showed gifted children disappoint on verbal tasks after watching violent cartoons. Hilgard, now an assistant professor at Illinois State University, told Retraction Watch in 2017 (when the paper was retracted) that the authors were “quite helpful.”

In December 2017, RUB “closed the case,” Ombudsman Eysel said. “We came to the well-grounded conclusion to apply the rule of protection of the honest whistleblower,” he told me. Markey and Elson have reposted the timeline of what happened to their website without attaching the documents they received from OSU.

Although the anxiety of the OSU complaint has now dissipated, Elson is left with another worry: If this paper by such a prominent lab about a topic of major public importance is flawed, what else is? “This was one paper—How many other papers are there out there that you can’t trust?” he told me. “What about all of these labs that are not as well known? Where not as many prying eyes are looking on? That worries me.”

Bushman is still frequently quoted in the news about the roots of teen violence—which is awkward for Markey, who says he gets upset when any researchers use data gathered in a controlled lab environment to try to explain something as multifactorial as school shootings. “Any scholar who makes that leap is making errors,” Markey told me.

The whole experience has been a bit surreal, Markey said. Though they were eventually vindicated, Markey and Elson paid a price for telling the world what had happened.

“At the end of the day, we weren’t saying anything that wasn’t true,” Elson told me. But simply ridding the literature of one problematic paper took countless hours of emails and discussions, not to mention the anxiety of dealing with the OSU complaint. Bringing out the truth about the shoddy study, “Took more time than actual research projects that we had,” he said. “It probably took more time than the actual study.”

For Markey, watching his friend and colleague live under the shadow of a complaint for a year was difficult, and not worth it. “At the end of the day, our goal was to remove bad science. And we achieved that goal. But I have no desire to ever do this again,” Markey said. “Because all it got me was grief.”