Baker - Closing the Loop
Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop”
Summary Table
| PIs | Ryan Baker |
| Other Contributors | Milan Desai |
| Study Start Date | Fall, 2010 |
| Study End Date | Spring, 2011 |
| LearnLab Site | Hopewell HS |
| LearnLab Course | Algebra I |
| Number of Students | TBD |
| Total Participant Hours | TBD |
| Data available in DataShop | TBD
|
Abstract
This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC (Baker_Choices_in_LE_Space) (Baker et al, 2009), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1 -- full details on this model are given on the page Baker_Choices_in_LE_Space) was created, and used to study why some tutor lessons are gamed more than others in the Algebra tutor. The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.
In this study, we will choose a lesson from the Algebra tutor that is highly gamed (CTA 12: Systems of Equations A), and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.
Background & Significance
In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as Help Abuse (Aleven & Koedinger, 2001), and Systematic Guessing (Baker et al, 2004). This pair of strategies is referred to as Gaming the system (Baker et al, 2004). Gaming the system is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004).
Recent work has indicated that a variety of aspects of cognitive tutor lessons are predictive of greater quantities of gaming (Baker et al, 2009). In particular, gaming is predicted by (boldface indicates a particularly strong relationship -- r2>0.15)
- Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest
- Not immediately apparent what icons in toolbar mean
- The lesson is not an equation-solver unit
- The location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated
- The same number being used for multiple constructs
- Reading hints does not positively influence performance on future opportunities to use skill
- Proportion of hints in each hint sequence that refer to abstract principles
- Hints do not give directional feedback such as “try a larger number”
- Hint requests that student perform some action
Historically, work to address gaming the system has either attempted to eliminate gaming behavior by making it more difficult to game (for instance, by putting delays between hints), or has attempted to detect gaming automatically and respond to it. The first approach appears to lead students to find new gaming strategies (Murray & VanLehn, 2005), and the second approach has led to systems that reduce gaming (Arroyo et al, 2007; Baker et al, 2006; Roll et al, 2007; Walonoski & Heffernan, 2006) and improve learning (Arroyo et al, 2007; Baker et al, 2006), but has typically been time-consuming and difficult to scale.
Within this project, we propose to use the findings from Baker et al (2009), which found tutor lesson features associated with gaming, as design principles for how to develop tutor lessons that students do not choose to game, in a completely unnoticeable way.
This study will also give us the opportunity to make a contribution to another area of research – recent work has suggested that interest-increasing text may reduce the transferability of knowledge (Kaminski et al, 2009). This may form a trade-off, where interest-increasing text reduces gaming (improving learning) but reduces the transferability of the learning. We will investigate this question with conditions that both include interest-increasing text, and lack interest-increasing text, but are identical in all other ways.
A third potential benefit of this study is in elucidating the link between these aspects of tutors and gaming behavior. We hypothesize that the mediating link is via the affective states of boredom and confusion. We will validate this hypothesis by observing student affect in each condition of the study.
Glossary
Computational Modeling and Data Mining
Hypotheses
- H1
- Re-designing a tutor lesson to eliminate features shown to be associated with gaming (see above), will result in lower gaming and better learning
- H2
- The tutor lesson features associated with gaming(see above) influence gaming by increasing the incidence of boredom and confusion
- H3
- Adding interest-increasing text to a tutor lesson (one such feature) will reduce gaming and improve learning, but potentially at the cost of lower transfer
Independent Variables
Three conditions will be compared:
- An unmodified version of a highly-gamed lesson in Cognitive Tutor Algebra (CTA 12: Systems of Equations A)
- A modified version of the same lesson
- Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
- Fewer help messages that are wholly abstract in nature
- A second modified version of the same lesson
- Adding interest-increasing extraneous text to the scenario
- Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
- Fewer help messages that are wholly abstract in nature
Dependent Variables
- Incidence of gaming behavior (measured via quantitative field observations -- cf. Baker et al, 2004)
- Incidence of boredom and confusion (measured via quantitative field observations -- cf. Rodrigo et al, 2007)
- Learning of domain skills and concepts (measured pre-post)
- Transfer and preparation for future learning (measured at post-test)
Planned Experiments
A comparison of the three conditions will be conducted in an in-vivo school study. We will control for time, and assign students randomly to conditions.
Explanation
Further Information
Connections
Annotated Bibliography
References
Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58
Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.
Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A., Mahadevan, S., and Woolf. B.P. (2007) Repairing Disengagement with Non- Invasive Interventions. Proceedings of the 13h International Conference on Artificial Intelligence in Education, 195-202.
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S.E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401.
Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.pdf
Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.
Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.
Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.
Kaminski, J.A., Sloutsky, V.M., Heckler, A. (2009) Transfer of Mathematical Knowledge: The Portability of Generic Instantiations. Child Development, 3 (3), 151-155.
Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While Providing Proactive Help. Proc. of the International Conference on Artificial Intelligence in Education, 887-889.
Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. pdf
Roll, I., Aleven, V., McLaren, B.M., and Koedinger, K.R. (2007) Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of the 13th International Conference on Artificial Intelligence in Education, 203-210.
Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.
Walonoski, J.A., Heffernan, N.T. (2006) Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 722-724.
