Using the RITE method to learn: testing the Mars game prototype. [i]

GUEST COLUMN | by Barbara Freeman

The purpose of the Mars Game project is to build a game that is ‘fun’ (meaning that it is immersive and that students want to play it), and to evaluate the potential of a digital learning game to effectively teach STEM content. This is the first in a series of research studies in which we are testing the Mars game with ninth and tenth grade students for whom the game is intended.

One student captured the reason so many game designers and researchers want to prove that aspects of gaming can enhance learning.

The gameplay takes students and transports them to Mars, where they crash land on the inhospitable planet where they need to work through a series of programming and mathematical challenges to stay alive. The math challenges are derived from the Common Core State Standards.

In this formative study, we tested the Mars game prototype to understand students’ initial reactions, to see if the developers were on the right track, and to make modifications to the game that would improve students’ experience moving forward. We used the RITE (Rapid Iterative Testing and Evaluation) Method[ii], a form of usability testing in which the researchers and the game developers work collaboratively with the end-users to identify problems and then work together to make rapid iterative changes to the game and confirm the effectiveness of those changes through further testing with a different set of students. Fifteen students of diverse backgrounds participated in the study. Students completed two levels of the game and were asked to think aloud as they played to describe their experience. Eighty percent of the issues were identified within the first day of testing. With the game developers working late into the night and early morning they were able to make rapid changes to the game, which were put before a different set of students to understand if these initial fixes resolved the problems that the students identified. This intense cycle of testing, fixing, and validating was repeated over five days.

By jointly working through the RITE process, the researchers and developers gained a shared understanding of the major issues and prioritized the changes that needed to be made to eliminate the “game stoppers” (so to speak). Most importantly, an analysis of the surveys[iii] that students were asked to complete showed students’ ease of using the Mars Game going up and their frustration going down over the course of the week. We also garnered initial evidence that showed the power of engagement with learning that emanates from gameplay. Even on day one, as students struggled to figure out how to use Blockly—Google’s visual programming language—to navigate the Mars terrain and the challenges of repairing a crashed spacecraft, the game could be seen to exert its magic. Once the tasks were mastered, students asked if there were more levels that they could play.

The magic appears to result from the problem-solving pull of the game with its embedded instructional challenges. Even though math and programming concepts are embedded in the game mechanics, some students said, “(the game) didn’t feel like math.” We think that’s a good thing, because the game is able to present concepts to students in a manner that is not intimidating to students.

For many, this was the first time that they had written a computer program, and they didn’t even realize that this was what they were doing. They were surprised and delighted when we pointed out to them that they had written their first ever program to get the rover to move during play.

One student captured the reason so many game designers and researchers want to prove that aspects of gaming can enhance learning: “at first you struggle, it is confusing, so you play around and get closer and closer to your goal and it makes you want to try again and again, and then you get it and you are happy.”

Our observations indicate that the experience was most intense for those students who were working within their zone of proximal development[iv], while those students who already understood the game’s content were least absorbed. We also learned that it did not take much to jolt students out of their immersion in the game environment. For example, many students withdrew when they heard the game voiceover mention the word “function.”

We came to understand that our mandate as game researchers and developers was to find a judicious balance that provides adequate supports without weakening the problem-solving pull of the game or students’ immersion in the virtual world. The game provides students with an experience that requires them to solve difficult yet tractable problems. Students work through the game’s challenges through exploration and experimentation, often collaboratively or with hints provided by the game, but without anybody explicitly telling them what to do. Through the process of solving these problems, students come to more deeply understand the material.

The fun is there: It lies in the problem-solving challenge; in the “aha” moment when students realize that they were actually coding. Next we will tell them they were also learning math!

—

[i] Barbara Freeman, Kevin Dill, Leslye Arsht, Kevin Oden, Mark Torpey, & Juan Benito

[ii] Medlock, M.C. Wixon, D., Terrano, M., & Romero, R. (2002). Using the RITE Method to improve products: A definition and a case study. Usability Professionals Association. Orlando, FL.

[iii] Surveys included a student self-assessment, the NASA Task Load Index, and the System Usability Scale.

[iv] Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press.

—

Barbara Freeman, Ph.D., is the principal research investigator on the Mars Game project, which was funded by the Advanced Distributed Learning Colab of the Department of Defense. She is a consultant and visiting scholar at U.C. Berkeley, and has over 25 years of domestic and international experience in technology, education, research and development, analytics, and risk management, and has established businesses and products for Blue Chip clients and government agencies globally. This article was co-authored by other members of the RITE research and development team, Kevin Dill (chief developer), Leslye Arsht, Kevin Oden, Mark Torpey, and Juan Benito.