Enabling Technology Benefits
In addition to DataShop mentioned above, LearnLab’s other enabling technologies have been a benefit to researchers around the world as well as LearnLab researchers (see right side of FIgure 6). Five LearnLab courses have made use of the Cognitive Tutor Authoring Tools (CTAT) in the development of core course material, experiments, or on-line assessments. CTAT is free to download from ctat.pact.cs.cmu.edu. CTAT was developed using empirical Human-Computer Interaction research methods and research studies have demonstrated its great value in making it possible for non-programmers to create online tutoring systems that provide the benefits of intelligent tutoring systems (Aleven, Mclaren, Sewall, & Koedinger, 2009, cited by 224). Instead of taking some 200 hours to develop an hour of intelligent tutor instruction, CTAT substantially reduces development cost by a factor of 4-8 times. CTAT has had major impact. Since 2006, CTAT has been downloaded 40,526 times and the CTAT website has drawn over 6.1 million hits from over 490,000 unique visitors. CTAT has also been used in other tutor development, for instance, for Genetics, fractions, collaborative problem solving, and middle school math. A recent paper reviews 18 tutors built with CTAT that have been used in real educational settings (Aleven et al., 2016), many with substantial pre/post learning gains. CTAT-built tutors have been used by approximately 44,000 students and account for about 40% of the data sets in DataShop (Aleven et al., 2016).
Reflecting broad impact and sustainability of center products, CTAT has been used in number of separately funded projects. The AdaptErrEx project (PI Bruce McLaren), funded by the U.S. Department of Education, developed materials with CTAT to support learning of decimals through erroneous examples in use by more than 400 students per school year (McLaren, Adams, & Mayer, 2015). A Genetics Problem-Solving Tutor implemented with CTAT by Albert Corbett and colleagues has been deployed at 14 colleges and universities and piloted at 3 LearnLab high schools. This tutor consists of 17 modules with 135 total problems that span a broad range of topics in genetics. An hour of problem solving has yielded an average learning gain of 18 points on a scale of 100, equivalent to almost two letter grades (Corbett, Kauffman, Maclaren & Wagner, 2010). MathTutor, an IES-funded project and website for middle-school students to learn mathematics through CTAT tutors (Aleven, McLaren & Sewall, 2009), has also seen extensive use. Through 2014, MathTutor itself was used by over 9,000 students, and together with other sites using its technology over 27,000 students logged over 61,200 student hours.
TagHelper (Rose et al., 2008, cited by 242) is a tool for the design of studies and interventions. It was designed to enable developers to develop filters than can “listen in” on student conversations or other natural language input from students, such as short answers to open ended questions. Thus, rather than being a tool for authoring interventions that offer interactive instruction themselves, as CTAT and TuTalk do, TagHelper is a tool used for monitoring student behavior in search of key events that can be used to assess how closely it does or does not conform to expected standards, so that it is possible to offer educational interventions to students on an as-needed basis, rather than treating all students and all student groups as though they have identical needs.
LightSIDE, a successor to TagHelper and a key LearnLab enabling technology, has been downloaded thousands of times by researchers across the world, and was highlighted as an Editor’s Choice in the June 2012 issue of Science. It allows users without extensive machine learning backgrounds to use automated analysis for their collected text or transcript data. Several day- and week-long workshops have been held in the summers of 2010-2012. In the spring of 2012, LightSIDE participated as an invited vendor in the Automated Student Assessment Prize along with eight commercial vendors including Pearson, McGraw-Hill, and ETS. The results of the study, which evaluated the ability of machine learning systems to automatically grade short student essays, showed that all competitors performed at a high level, matching or approaching human reliability. LightSIDE is the only tool from that competition which is freely available and open source for research purposes. The study was covered in many press venues including Education Week, the New York Times, and NPR. Based on this work, Elijah Mayfield, a LearnLab PhD student, started spin-off company, LightSide Labs, which was acquired in 2014 by Turnitin.
Bazaar is another freely-available LearnLab enabling technology. As a framework for authoring and deploying multiparty conversational tutorial agents, Bazaar (along with its predecessors, Basilica and TuTalk) has served as the backbone of the Social Communication thrust’s recent experiments in online group learning. Modules written for Bazaar agents include support tutorial dialogue along structured lines of reasoning, responding to student turns based on real-time automatic classification (using models built in LightSIDE), dynamic facilitation of student interaction to support social cohesion, participation, and academically productive talk. Bazaar was introduced in the CSCL track at the IPTSE Winter School in 2011, and the LearnLab summer school in 2012. TuTalk and Basilica have been the focus of CSCL tracks at the Winter and Summer Schools since the inception of LearnLab.