Education and publishing giant Pearson is drawing criticism after using its software to experiment on over 9,000 math and computer science students across the country. In a paper presented Wednesday at the American Association of Educational Research, Pearson researchers revealed that they tested the effects of encouraging messages on students that used the MyLab Programming educational software during 2017's spring semester.
Titled “Embedding Research-Inspired Innovations in EdTech: An RCT of Social-Psychological Interventions, at Scale,” the study placed 9,000 students using MyLab Programming into three groups, each receiving different messages from the software as they attempted to solve questions. Some students received “growth-mindset messages,” while others received “anchoring of effect” messages. (A third control group received no messaging at all.) The intent was to see if such messages encouraged students to solve more problems. Neither the students nor the professors were ever informed of the experiment, raising concerns of consent.
The “growth mindset messages” emphasized that learning a skill is a lengthy process, cautioning students offering wrong answers not to expect immediate success. One example: “No one is born a great programmer. Success takes hours and hours of practice.” “Anchoring of effect” messages told students how much effort is required to solve problems, such as: “Some students tried this question 26 times! Don’t worry if it takes you a few tries to get it right.”
As Education Week reports, the interventions offered seemingly no benefit to the students. Students who received no special messages attempted to solve more problems (212) than students in either the growth-mindset (174) or anchoring groups (156). The researchers emphasized this could have been due any of a variety of factors, as the software is used differently in different schools. However, educators who spoke to Education Week were understandably more alarmed by Pearson placing thousands of students in A/B testing for its products.
“It’s concerning that forms of low-level psychological experimentation to trigger certain behaviors appears to be happening in the ed-tech sector, and students might not know those experiments are taking place,” Ben Williamson, a professor at the University of Stirling, told the publication.
In a statement to Gizmodo, Pearson emphasized that the experiment was “an effort to improve student success in higher education courseware”:
The Education Week article, “Pearson Tested ‘Social-Psychological’ Messages in Learning Software, With Mixed Results,” mischaracterizes a relatively minor product update by extracting technical language from research that, when taken out of context, and paired with words like “experiment,” conveys a malicious intent.
As with any changes to a product, we evaluate the introduction of changes to determine if they require additional ethical or legal review or consultation. In this case, the introduction of feedback messages about how to improve student success, was determined to be a part of normal educational practice.
The experiment has troubling overlaps with at least one of Facebook’s many overlapping privacy cataclysms. In 2014, Facebook experimented on almost 700,000 users, changing what they saw in their news feeds and recording the impact on their moods. Users were never told their moods were the subject of a psychological experiment and critics called that experiment unethical.
“Randomized control trials like this, at scale and embedded into widely used commercial products, are a valuable approach for improving learner outcomes in a rigorous and iterative way, while also contributing to the burgeoning literature on social-psychological interventions,” Pearson’s study reads.
Update 4/19/18: After publication, Gizmodo spoke with Pearson’s Vice President of Learning Research and Design, Kristen Dicerbo. Dicerbo disagreed with both Gizmodo and Education Week’s framing, saying the My Programming Lab messages weren’t psychological experiments, but product tests.
“There’s often this idea that an experiment is where you’re manipulating people,” she told Gizmodo, “or manipulating how they’re thinking in a setting where were trying to think about confirming or testing our hypotheses. Here what we’re really doing is thinking about product improvement [and] does this improve student outcome? We’re thinking about, ‘Does this help students learn?’ and ‘Can we design it in a way that can help students learn?’ specific to the design of our product.”
Such product tests don’t require informed consent, Dicerbo said, referencing HHS protocol. She also noted that other companies producing learning software do similar tests, referencing a Fast Company write-up of DuoLingo’s product improvement tests, where different users are shown different messages. While DuoLingo doesn’t frame their tests as “social-psychological,” Dicerbo stressed the specific context of the AERA conference.
“When we say [social-psychological] in the context of a research paper, what we’re saying is that some of the ideas come from the social psychological research literature,” she said. “This idea of how we can give encouragement to students after they’ve been unsuccessful has been studied in the social psychological research.”
Pearson’s research was peer-reviewed before acceptance to the conference and Dicerbo is considering publishing the findings in a scientific journal.
“We’re not saying that we’re attempting to change students’ social or emotional functions. I think out in the wider world when people hear that... they think we’re trying to change how students interact with people.”
Additionally, Gizmodo removed a reference to “minors” in the original story to describe the students involved in testing. As a Pearson spokesperson noted, most freshman college students are over 18.
[Edweek h/t Kate Crawford]