Difference between revisions of "Gaming the system"
(added results from PSLC project) |
m (Reverted edits by Shelaybarra (Talk); changed back to last version by Mbett) |
||
| (10 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
Baker et al (2006) defines gaming the system as "Attempting to succeed in an interactive learning environment by exploiting properties of the system rather than by learning the material". | Baker et al (2006) defines gaming the system as "Attempting to succeed in an interactive learning environment by exploiting properties of the system rather than by learning the material". | ||
| − | Within intelligent tutoring systems such as Cognitive Tutors, this is usually done by | + | Within intelligent tutoring systems such as Cognitive Tutors, this is usually done by systematic guessing, where a student systematically tries a set of possible answers (example: 1, 2, 3, 4, 5, ... 38) or hint abuse, drilling through hints at high speed to obtain the answer (Aleven & Koedinger, 2000). |
Gaming has been observed in other types of learning environments as well, including educational games (Miller, Lehman, & Koedinger, 1999; Magnussen & Misfeldt, 2004; Rodrigo et al, 2007), simulation environments (Rodrigo et al, 2007), and graded-participation newsgroups (Cheng & Vassileva, 2005). | Gaming has been observed in other types of learning environments as well, including educational games (Miller, Lehman, & Koedinger, 1999; Magnussen & Misfeldt, 2004; Rodrigo et al, 2007), simulation environments (Rodrigo et al, 2007), and graded-participation newsgroups (Cheng & Vassileva, 2005). | ||
| + | |||
| + | It has been repeatedly shown that students who game the system have poorer learning than non-gaming students with comparable pre-test scores (Baker et al, 2004, 2006; Walonoski & Heffernan, 2006a). (One exception is when students drill through hints, and then self-explain them -- Shih et al, 2008; another exception is when students game time-consuming material they already know -- Baker, Corbett & Koedinger, 2004). | ||
Gaming the System has been shown to be associated with the affective experiences of boredom and confusion (Rodrigo et al, 2007) -- in specific, a student who experiences either of these two affective states is significantly more likely to be gaming the system shortly afterwards. Frustration, though previously found to be associated with gaming (Baker et al, 2008), appears to co-occur with gaming behavior rather than preceding it. | Gaming the System has been shown to be associated with the affective experiences of boredom and confusion (Rodrigo et al, 2007) -- in specific, a student who experiences either of these two affective states is significantly more likely to be gaming the system shortly afterwards. Frustration, though previously found to be associated with gaming (Baker et al, 2008), appears to co-occur with gaming behavior rather than preceding it. | ||
| Line 9: | Line 11: | ||
A variety of stable or semi-stable student characteristics have been studied in relation to gaming the system (e.g. Arroyo & Woolf, 2005; Baker et al, 2008; Beal, Qu, & Lee, 2009); however, these characteristics have generally been found to have weak correlations with gaming, at best. Some characteristics found to be significantly associated with gaming include negative attitudes towards computers, the learning software, and mathematics. Performance goals and anxiety have been repeatedly found to have no correlation to gaming (Baker et al, 2008). | A variety of stable or semi-stable student characteristics have been studied in relation to gaming the system (e.g. Arroyo & Woolf, 2005; Baker et al, 2008; Beal, Qu, & Lee, 2009); however, these characteristics have generally been found to have weak correlations with gaming, at best. Some characteristics found to be significantly associated with gaming include negative attitudes towards computers, the learning software, and mathematics. Performance goals and anxiety have been repeatedly found to have no correlation to gaming (Baker et al, 2008). | ||
| − | Recent results from PSLC project | + | Recent results from PSLC project [[Baker_Choices_in_LE_Space | How Content and Interface Features Influence Student Choices Within the Learning Space]] indicate that differences between tutor lessons explain much more of the variance in how much students choose to game, than individual differences between students. This finding was obtained through an ANOVA conducted at each of these two levels, and was replicated in both the middle school Cognitive Tutor (precursor to Bridge to Algebra) (Baker, 2007), and the Algebra Cognitive Tutor (paper in preparation). |
Further data mining analysis (paper in preparation) using the [[CTLVS | Cognitive Tutor Lesson Variation Space (CTLVS)]] showed that students game the system more on lessons which have features which are likely to increase student confusion (including hints which do not lead any students to better performance, reference to abstract principles in hints, whether the toolbar is unclear, and the same number being used for multiple constructs) and boredom (including time-consuming problem steps and the lack of interest-increasing text in problem statements). These results conform well to the previous evidence on which affective states are associated with gaming. | Further data mining analysis (paper in preparation) using the [[CTLVS | Cognitive Tutor Lesson Variation Space (CTLVS)]] showed that students game the system more on lessons which have features which are likely to increase student confusion (including hints which do not lead any students to better performance, reference to abstract principles in hints, whether the toolbar is unclear, and the same number being used for multiple constructs) and boredom (including time-consuming problem steps and the lack of interest-increasing text in problem statements). These results conform well to the previous evidence on which affective states are associated with gaming. | ||
| − | + | [[Scooter the Tutor]] is a software agent who responds to gaming the system with emotional expressions and supplementary exercises (Baker et al, 2006). Scooter was associated with significantly reduced gaming, and significantly improved learning for gaming students (specifically those who received supplementary exercises). Scooter was built on top of the gaming detector, software validated to automatically detect gaming in running Cognitive Tutors (Baker et al, 2008). | |
| − | + | ||
=== PSLC Studies Involving Gaming the System === | === PSLC Studies Involving Gaming the System === | ||
*[[Baker_Choices_in_LE_Space | How Content and Interface Features Influence Student Choices Within the Learning Space (Baker, Corbett, Koedinger, & Rodrigo)]] | *[[Baker_Choices_in_LE_Space | How Content and Interface Features Influence Student Choices Within the Learning Space (Baker, Corbett, Koedinger, & Rodrigo)]] | ||
| + | * [[Baker - Closing the Loop]] | ||
| + | * [[Baker - Building Generalizable Fine-grained Detectors]] | ||
| + | |||
| + | === See Also === | ||
| + | Ryan Baker's [http://www.joazeirodebaker.net/ryan/gaming.html webpage on gaming the system]. | ||
=== Bibliography === | === Bibliography === | ||
| − | + | * Aleven, V., Koedinger, K.R. (2000)Limitations of Student Control: Do Students Know When They Need Help? Proceedings of the 5th International Conference on Intelligent Tutoring Systems, 292-303. | |
| − | * Baker, R.S.J.d. (2007) Is Gaming the System State-or-Trait? Educational Data Mining Through the Multi-Contextual Application of a Validated Behavioral Model. Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, 76-80. | + | * Baker, R.S.J.d. (2007) Is Gaming the System State-or-Trait? Educational Data Mining Through the Multi-Contextual Application of a Validated Behavioral Model. Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, 76-80. [http://www.cs.cmu.edu/~rsbaker/B2007B.pdf pdf] |
| − | * Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. ( | + | * Baker, R.S., Corbett, A.T., Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 531-540. [http://www.cs.cmu.edu/~rsbaker/BCK2004MLFinal.pdf pdf] |
| − | * Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006). Adapting to When Students Game an Intelligent Tutoring System. 8th International Conference on Intelligent Tutoring Systems, 392-401. [http://www.joazeirodebaker.net/ryan/Baker175.pdf pdf] | + | * Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (2008) Developing a Generalizable Detector of When Students Game the System User Modeling and User-Adapted Interaction, 18, 3, 287-314. [http://www.joazeirodebaker.net/ryan/USER475.pdf pdf] |
| + | * Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S. E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006). Adapting to When Students Game an Intelligent Tutoring System. 8th International Conference on Intelligent Tutoring Systems, 392-401. [http://www.joazeirodebaker.net/ryan/Baker175.pdf pdf] | ||
* Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. ACM CHI 2004: Computer-Human Interaction, 383-390. [http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf] | * Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. ACM CHI 2004: Computer-Human Interaction, 383-390. [http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf] | ||
* Baker, R.S.J.d., Walonoski, J.A., Heffernan, N.T., Roll, I., Corbett, A.T., Koedinger, K.R. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224. [http://www.joazeirodebaker.net/ryan/BWHRKC-JILR-draft.pdf pdf] | * Baker, R.S.J.d., Walonoski, J.A., Heffernan, N.T., Roll, I., Corbett, A.T., Koedinger, K.R. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224. [http://www.joazeirodebaker.net/ryan/BWHRKC-JILR-draft.pdf pdf] | ||
| Line 33: | Line 41: | ||
* Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. | * Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. | ||
[http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf] | [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf] | ||
| + | * Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126. [http://www.educationaldatamining.org/EDM2008/uploads/proc/12_Shih_35.pdf pdf] | ||
* Walonoski, J.A., Heffernan, N.T. (2006a) Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 382-391. | * Walonoski, J.A., Heffernan, N.T. (2006a) Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 382-391. | ||
* Walonoski, J.A., Heffernan, N.T. (2006b) Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 722-724. | * Walonoski, J.A., Heffernan, N.T. (2006b) Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 722-724. | ||
Latest revision as of 20:32, 2 September 2011
Baker et al (2006) defines gaming the system as "Attempting to succeed in an interactive learning environment by exploiting properties of the system rather than by learning the material".
Within intelligent tutoring systems such as Cognitive Tutors, this is usually done by systematic guessing, where a student systematically tries a set of possible answers (example: 1, 2, 3, 4, 5, ... 38) or hint abuse, drilling through hints at high speed to obtain the answer (Aleven & Koedinger, 2000).
Gaming has been observed in other types of learning environments as well, including educational games (Miller, Lehman, & Koedinger, 1999; Magnussen & Misfeldt, 2004; Rodrigo et al, 2007), simulation environments (Rodrigo et al, 2007), and graded-participation newsgroups (Cheng & Vassileva, 2005).
It has been repeatedly shown that students who game the system have poorer learning than non-gaming students with comparable pre-test scores (Baker et al, 2004, 2006; Walonoski & Heffernan, 2006a). (One exception is when students drill through hints, and then self-explain them -- Shih et al, 2008; another exception is when students game time-consuming material they already know -- Baker, Corbett & Koedinger, 2004).
Gaming the System has been shown to be associated with the affective experiences of boredom and confusion (Rodrigo et al, 2007) -- in specific, a student who experiences either of these two affective states is significantly more likely to be gaming the system shortly afterwards. Frustration, though previously found to be associated with gaming (Baker et al, 2008), appears to co-occur with gaming behavior rather than preceding it.
A variety of stable or semi-stable student characteristics have been studied in relation to gaming the system (e.g. Arroyo & Woolf, 2005; Baker et al, 2008; Beal, Qu, & Lee, 2009); however, these characteristics have generally been found to have weak correlations with gaming, at best. Some characteristics found to be significantly associated with gaming include negative attitudes towards computers, the learning software, and mathematics. Performance goals and anxiety have been repeatedly found to have no correlation to gaming (Baker et al, 2008).
Recent results from PSLC project How Content and Interface Features Influence Student Choices Within the Learning Space indicate that differences between tutor lessons explain much more of the variance in how much students choose to game, than individual differences between students. This finding was obtained through an ANOVA conducted at each of these two levels, and was replicated in both the middle school Cognitive Tutor (precursor to Bridge to Algebra) (Baker, 2007), and the Algebra Cognitive Tutor (paper in preparation).
Further data mining analysis (paper in preparation) using the Cognitive Tutor Lesson Variation Space (CTLVS) showed that students game the system more on lessons which have features which are likely to increase student confusion (including hints which do not lead any students to better performance, reference to abstract principles in hints, whether the toolbar is unclear, and the same number being used for multiple constructs) and boredom (including time-consuming problem steps and the lack of interest-increasing text in problem statements). These results conform well to the previous evidence on which affective states are associated with gaming.
Scooter the Tutor is a software agent who responds to gaming the system with emotional expressions and supplementary exercises (Baker et al, 2006). Scooter was associated with significantly reduced gaming, and significantly improved learning for gaming students (specifically those who received supplementary exercises). Scooter was built on top of the gaming detector, software validated to automatically detect gaming in running Cognitive Tutors (Baker et al, 2008).
PSLC Studies Involving Gaming the System
- How Content and Interface Features Influence Student Choices Within the Learning Space (Baker, Corbett, Koedinger, & Rodrigo)
- Baker - Closing the Loop
- Baker - Building Generalizable Fine-grained Detectors
See Also
Ryan Baker's webpage on gaming the system.
Bibliography
- Aleven, V., Koedinger, K.R. (2000)Limitations of Student Control: Do Students Know When They Need Help? Proceedings of the 5th International Conference on Intelligent Tutoring Systems, 292-303.
- Baker, R.S.J.d. (2007) Is Gaming the System State-or-Trait? Educational Data Mining Through the Multi-Contextual Application of a Validated Behavioral Model. Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, 76-80. pdf
- Baker, R.S., Corbett, A.T., Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 531-540. pdf
- Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (2008) Developing a Generalizable Detector of When Students Game the System User Modeling and User-Adapted Interaction, 18, 3, 287-314. pdf
- Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S. E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006). Adapting to When Students Game an Intelligent Tutoring System. 8th International Conference on Intelligent Tutoring Systems, 392-401. pdf
- Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. ACM CHI 2004: Computer-Human Interaction, 383-390. pdf
- Baker, R.S.J.d., Walonoski, J.A., Heffernan, N.T., Roll, I., Corbett, A.T., Koedinger, K.R. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224. pdf
- Beal, C. R., Qu, L., & Lee, H. (2009). Mathematics motivation and achievement as predictors of high school students' guessing and help-seeking with instructional software. Journal of Computer Assisted Learning.
- Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.
- Magnussen, R., Misfeldt, M. (2004) Player Transformation of Educational Multiplayer Games. Proceedings of Other Players. Available at http://www.itu.dk/op/proceedings.htm
- Miller, C.S., Lehman, J.F., Koedinger, K.R. (1999) Goals and learning in microworlds - An exploration. Cognitive Science, 23 (3), 305-336.
- Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While Providing Proactive Help. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 887-889.
- Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152.
- Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126. pdf
- Walonoski, J.A., Heffernan, N.T. (2006a) Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 382-391.
- Walonoski, J.A., Heffernan, N.T. (2006b) Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 722-724.
