A passionate educator’s plea for quality lights up the edtech space. 

INTERVIEW | by Victor Rivero

For a long time, Karen Mahon had been really bothered by the poor quality of educational apps in the apps stores and the fact that it’s pretty much impossible to tell what is good and what’s not when browsing through app stores. “I found that I kept complaining about it in my blog and in conversations that I was having,” she says. “I kept asking, ‘Why doesn’t somebody create a review service based on what we know works in education?’” She was referring to the decades of empirical research examining how people learn best. “Then one day I woke up and thought, ‘Hmm, I’m somebody!’ So I decided that I should make the effort to do the thing that I thought was missing instead of just complaining about it,” she laughs. “And I was fortunate to have a group of friends and colleagues who wanted to join me in that effort!” Today, Karen is president and founder of Balefire Labs, Inc., a provider of non-biased, objective reviews that show which apps help children learn. Here she discusses her thoughts on a range of relevant items, including the origin of their evaluation criteria, where she sees the educational apps sector headed in the near future, a brand-new partnership with another player in the market, and an exciting new program they’ve just launched.

Victor: What holes or gaps in education were you trying to fill? 

Karen: In short, we are trying to provide a science-based, objective and unbiased source of educational app reviews. Our criteria are based on scientific research about learning and we publish, on our website, research citations to support those criteria. Our reviews are objective in that we evaluate all of the apps according to that same set of research-based criteria, such that apps can be compared and contrasted directly with each other. And finally, we are unbiased – we take no money from developers or corporations that might have a vested interest in the outcomes of our reviews.

Victor: Are there any similar services in other industries that you considered as somewhat of a model? Which ones? 

Karen: The service that we aspire to be like is Consumer Reports. Their reviews are clear and objective and conducted by trained experts, not users. Products in a given category are all evaluated in the same way. And finally, there is no pay-for-play or advertising. Consumer Reports’ is a trusted resource with high credibility, as a result. We strive to have that same kind of reputation in the education technology space.

Victor: What do you think about other places that provide a similar service, such as edshelf, graphite or even certain sections of EdSurge?

Karen: We’re glad that there are other services in this space helping draw attention to the quality problem in education technology. And we think that with the huge proliferation of ed tech products, particularly apps, the market will get even more confusing in the next few years. We do think that our offering is unique in this space, however. The primary difference is in how we evaluate apps. The criteria that are in our rubric are based on results of empirical, published research from the past 50 years. We have combed the research to identify the instructional design and usability design features that positively impact learning outcomes. We make those criteria completely transparent to our customers and to developers, so that they understand the precise rationale behind the ratings of the apps on our site. No one else is doing that. In fact, most other educational app review services provide subjective, opinion-based reviews or they have evaluation categories such as “pedagogy,” that are not defined for the customer. Because our review criteria are so clearly and explicitly defined, we are able to apply them to all apps in a standard way, thus allowing direct comparison between apps. We can then show the results in easy-to-read charts that let subscribers compare and contrast apps at a glance. Even if educators don’t care about all of the same criteria that we think are important, the clarity of our process allows them to see exactly what features apps do and do not have and they can select apps based on their own preferences.

In addition to our review process being different from others in the space, we have a different business model. We ask educators and parents to pay a subscription fee for our service. Almost all of the other educational app review sites we know of are free to teachers and parents. But when a service is free you have to ask who is footing the bill — because someone definitely is. Usually it is developers or corporations with a vested interest in which apps teachers and parents buy. In fact, oftentimes developers must pay to have their apps reviewed or posted on those sites. We want customers to trust that we are unbiased in our reviews; that there is no conflict of interest for us because we are “clean” of developer money. We don’t think it’s fair that educators and parents only see apps of companies that can afford to buy access to customers. We charge the people who we work for: educators and parents. 

Victor: You’ve recently partnered with edshelf, a company that might have been a potential competitor. How did that process go, and how is it that it’s a mutually beneficial pairing, not only for both companies, but for parents and educators?  

Karen: At first blush, we might seem like competitors, but we’re actually quite complementary. edshelf includes a vast array of edtech resources on their site, not just mobile apps, and the reviews available are subjective user reviews. We think we can bring value to the edshelf users with a selection of our TopRated Apps that have gone through our rigorous, objective review process, conducted by trained evaluators. In addition, we really like what edshelf does in allowing users to create their own personal collections of resources on the site. We think a tool like that will be of interest to our customers at Balefire Labs. So we’re really excited about this relationship and looking forward to collaborations that help all teachers and parents!

Victor: There are a lot of very talented people on your team – how did you pull them all together? Through what various associations did you come together?

Karen: I’ve been really fortunate to work with a lot of great people in my career. But the group that is on this team has been hand-picked because of who they are and their fit with our company culture, not just what they bring to the table. In our company we value transparency, authenticity and respect, not just with each other, but with our customers, our developers and our competitors. The people on our Board of Directors and Advisory Board are so talented that they could easily work with anyone they wished. They’ve chosen Balefire Labs because we’re values-driven and are doing this to really make a difference for kids.

Victor: Ultimately, do you see that this will sort the wheat from the chaff and actually move the needle so to speak on improving the ed app landscape? 

Karen: That is definitely the goal. We see our service as helping not only teachers and parents, but also educational app developers in several ways. First, we hope to drive customer purchases toward those developers who are building great educational apps, thus allowing them to use their profits to build more good apps. Second, because our process is transparent, developers can see the precise areas in which their apps can be improved. Many apps could be greatly improved with small modifications. Finally, by becoming subscribers to our service, developers get a lot of great competitive analysis information and can see what subjects and age levels are ripe for new, high-quality apps. It’s a team effort to get a shift in this space and we hope to be collaborators with developers who are dedicated to providing really effective learning experiences for kids.

Victor: Broadly, what are your thoughts on education today? 

Karen: That is broad! I think education is in flux in many areas. The role of technology, distance learning, new kinds of assessment…all of these are hotly debated and all have the potential to significantly impact how kids are educated in our country and around the world. And I can’t predict how it will all turn out, but I do think it’s clear that technology in education isn’t going away and is becoming more common. After many years of expecting educational outcomes to change just by adding hardware products to the classroom, we, as a market, are finally getting more sophisticated in talking about truly meaningful learner performance with edtech tools. But it’s a process — we think we can help.

Victor: Your review criteria fully intends to arrive at standardization of reviews. Do you feel the criteria itself is still in motion? Will it move more as new technologies come into the fray, or is it classic and timeless?

Karen: The criteria themselves, as you know, are based on decades of learning research, so the core definitions of the criteria aren’t likely to change much. As new technologies become more common, however, some of the details may change. For example, we currently define a high rate of relevant learner responding as at least one touch per screen in an app. But as we see gesture technology coming on, or even eye gaze, we’ll need to reexamine the definitions around how the learner is responding. Additionally, we do expect to add criteria, as we go forward, to include things such as Universal Design features, alignment to the Common Core State Standards, and other information that is likely to be useful to customers.

Victor: Will you have some sort of certification or rating system for solutions providers to use, such as a badge system that they might use to promote their authenticity in claims and usefulness, as rated by your system?

Karen: We just this week launched our Top Rated App badge program. The Top Rated App badge is available to developers of apps that earn an A or B grade, according to our criteria. To date, we have reviewed more than 1300 apps and only about seven percent of them have earned an A or B, so the Top Rated App badge is a real achievement. With this badge, customers will be able to identify the best educational apps easily when they visit app developers’ sites. So start looking for that!

Victor: Anything else you’d like to add or emphasize, anything we may have missed that you really wish to address here?

Karen: Thank you, Victor, for this opportunity to talk about what we’re doing at Balefire Labs, Inc. I really appreciate the opportunity!

—

Victor Rivero is the Editor in Chief of EdTech Digest. Are you an edtech leader, trendsetter or the creator of cool tools or apps for learning? There is still time to enter the 2014 EdTech Digest Awards Recognition Program until the final entry window closes on October 18, 2013. For full details and an entry form, write to: victor@edtechdigest.blog