Baker Choices in LE Space - Revision history

Ryan at 21:26, 5 December 2009

2009-12-05T21:26:18Z

← Older revision		Revision as of 21:26, 5 December 2009
Line 229:		Line 229:
	Baker, R.S.J.d. (2007b) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.		Baker, R.S.J.d. (2007b) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

	Baker, R.S.J.d. (~~accepted~~) Differences Between Intelligent Tutor Lessons, and the Choice to Go Off-Task. ~~To appear in~~ Proceedings of ~~EDM 2009~~.		Baker, R.S.J.d. (2009) Differences Between Intelligent Tutor Lessons, and the Choice to Go Off-Task. Proceedings of the 2nd International Conference on Educational Data Mining, 11-20.

	Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (~~in press a~~) Educational Software Features that Encourage and Discourage "Gaming the System". ~~To appear in~~ Proceedings of ~~AIED 2009~~.		Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

	Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]		Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Ryan: /* Connections to Other PSLC Studies */

2009-12-05T20:55:35Z

Connections to Other PSLC Studies

← Older revision		Revision as of 20:55, 5 December 2009
Line 216:		Line 216:
	=== Connections to Other PSLC Studies===		=== Connections to Other PSLC Studies===

	This study inspired and led to the upcoming Year 6 study, [[Baker Closing the Loop ~~on Gaming~~]].		This study inspired and led to the upcoming Year 6 study, [[Baker - Closing the Loop]].

	===Annotated Bibliography===		===Annotated Bibliography===

Ryan: /* References */

2009-05-16T01:49:43Z

References

← Older revision		Revision as of 01:49, 16 May 2009
Line 229:		Line 229:
	Baker, R.S.J.d. (2007b) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.		Baker, R.S.J.d. (2007b) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

	Baker, R.S.J.d. (accepted a) Differences Between Intelligent Tutor Lessons, and the Choice to Go Off-Task. To appear in Proceedings of EDM 2009.		Baker, R.S.J.d. (accepted) Differences Between Intelligent Tutor Lessons, and the Choice to Go Off-Task. To appear in Proceedings of EDM 2009.

	Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (in press a) Educational Software Features that Encourage and Discourage "Gaming the System". To appear in Proceedings of AIED 2009.		Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (in press a) Educational Software Features that Encourage and Discourage "Gaming the System". To appear in Proceedings of AIED 2009.

Ryan at 01:47, 16 May 2009

2009-05-16T01:47:50Z

← Older revision		Revision as of 01:47, 16 May 2009
Line 179:		Line 179:
	In the Middle School Tutor, lesson has 35 parameters and achieves an r-squared of 0.55. Student has 240 parameters and achieves an r-squared of 0.16. In the Algebra Tutor, lesson has 21 parameters and achieves an r-squared of 0.18. Student achieves an equal r-squared, but with 58 students; hence, lesson is a statistically better predictor because it achieves equal or significantly better fit with considerably fewer parameters.		In the Middle School Tutor, lesson has 35 parameters and achieves an r-squared of 0.55. Student has 240 parameters and achieves an r-squared of 0.16. In the Algebra Tutor, lesson has 21 parameters and achieves an r-squared of 0.18. Student achieves an equal r-squared, but with 58 students; hence, lesson is a statistically better predictor because it achieves equal or significantly better fit with considerably fewer parameters.

	We empirically grouped the 79 features of the CTLVS1.1 with Principal Component Analysis (PCA). We grouped the 79 features of the CTLVS1 into 6 ~~clusters~~. We then analyzed whether the correlation between these 6 ~~clusters and~~ the frequency of gaming the system was significant in any case.		We empirically grouped the 79 features of the CTLVS1.1 with Principal Component Analysis (PCA). We grouped the 79 features of the CTLVS1 into 6 factors. We then analyzed whether the correlation between these 6 factorsand the frequency of gaming the system was significant in any case.

	Of these 6 ~~clusters~~, one was statistically significantly associated with the choice to game the system, r2 = 0.29 (e.g. accounting for 29% of the variance in gaming), F(1,19)= 7.84, p=0.01. The ~~cluster~~ loaded strongly on eight features associated with more gaming:		Of these 6 factors, one was statistically significantly associated with the choice to game the system, r2 = 0.29 (e.g. accounting for 29% of the variance in gaming), F(1,19)= 7.84, p=0.01. The factor loaded strongly on eight features associated with more gaming:
	* 14: The same number being used for multiple constructs		* 14: The same number being used for multiple constructs
	* 23-inverse-direction: Reading hints does not positively influence performance on future opportunities to use skill		* 23-inverse-direction: Reading hints does not positively influence performance on future opportunities to use skill
Line 191:		Line 191:
	* 75: Hint requests that student perform some action		* 75: Hint requests that student perform some action

	In general, several of the features in this ~~cluster~~ appear to correspond to a lack of clarity in the presentation of the content or task (23-inverse, 40, 63-inverse), as well as abstractness (27) and ambiguity (14). Curiously, feature 71-inverse (the lack of a specific type of implementation flaw in hint messages, which would make things very unclear) appears to point in the opposite direction – however, this implementation flaw was only common in a single rarely gamed lesson, so this result is probably a statistical artifact.		In general, several of the features in this factor appear to correspond to a lack of clarity in the presentation of the content or task (23-inverse, 40, 63-inverse), as well as abstractness (27) and ambiguity (14). Curiously, feature 71-inverse (the lack of a specific type of implementation flaw in hint messages, which would make things very unclear) appears to point in the opposite direction – however, this implementation flaw was only common in a single rarely gamed lesson, so this result is probably a statistical artifact.

	Feature 53-inverse appears to represent a different construct – interestingness (or the attempt to increase interestingness). The fact that feature 53 was associated with less gaming whereas more specific interest-increasing features (features 46-52) were not so strongly related may suggest that it is less important exactly how a problem scenario attempts to increase interest, than it is important that the problem scenario has some content in it that is not strictly mathematical.		Feature 53-inverse appears to represent a different construct – interestingness (or the attempt to increase interestingness). The fact that feature 53 was associated with less gaming whereas more specific interest-increasing features (features 46-52) were not so strongly related may suggest that it is less important exactly how a problem scenario attempts to increase interest, than it is important that the problem scenario has some content in it that is not strictly mathematical.

	Taken individually, two of the constructs in this ~~cluster~~ were significantly (or marginally significantly) associated with gaming. Feature 53-inverse (text in the problem statement not directly related to the problem-solving task) was associated with significantly less gaming, r2 = 0.19, F(1,19) = 4.59, p = 0.04. Feature 40 (when it is not immediately apparent what icons in toolbar mean) was marginally significantly associated with more gaming, r2 = 0.15, F(1, 19)=3.52, p=0.08. The fact that other top features in the ~~cluster~~ were not independently associated with gaming, while the ~~cluster~~ as a whole was fairly strongly associated with gaming, suggests that gaming may occur primarily when more than one of these features are present.		Taken individually, two of the constructs in this factor were significantly (or marginally significantly) associated with gaming. Feature 53-inverse (text in the problem statement not directly related to the problem-solving task) was associated with significantly less gaming, r2 = 0.19, F(1,19) = 4.59, p = 0.04. Feature 40 (when it is not immediately apparent what icons in toolbar mean) was marginally significantly associated with more gaming, r2 = 0.15, F(1, 19)=3.52, p=0.08. The fact that other top features in the factor were not independently associated with gaming, while the factor as a whole was fairly strongly associated with gaming, suggests that gaming may occur primarily when more than one of these features are present.

	Two features that were not present in the significant ~~cluster~~ was statistically significantly associated with gaming: Feature 36, where the location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated, r2 = 0.20, F(1,19)=4.97, p=0.04. This feature, like many of those in the gaming-related ~~cluster~~, represents an unclear or confusing lesson.		Two features that were not present in the significant factor was statistically significantly associated with gaming: Feature 36, where the location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated, r2 = 0.20, F(1,19)=4.97, p=0.04. This feature, like many of those in the gaming-related factor, represents an unclear or confusing lesson. Also, Feature 79, whether or not the lesson was an equation solver unit, was statistically significantly better than chance, r2 = 0.30, F(1, 19)=8.55, p<0.01. Note, however, that although a lower amount of interesting text is generally associated with more gaming (Feature 53), equation-solver units (which have no text) have less gaming in general (Feature 79). This result may suggest that interest-increasing text is only beneficial (for reducing gaming) above a certain threshold -- alternatively, other aspects of the equation-solver units may have reduced gaming even though the lack of interesting-increasing text would generally be expected to increase it.

	When the gaming-related ~~cluster~~ and Feature 36 were included in a model together, ~~both~~ remain statistically significant, and the combined model explains 41% of the variance in gaming (e.g. r2 = 0.41).		When the gaming-related factor, Feature 36, and Feature 79, were included in a model together, all remain statistically significant, and the combined model explains 56% of the variance in gaming (e.g. r2 = 0.55).

	Five other features that were not strongly loaded in the significant ~~cluster~~ were marginally associated with gaming. None of these other features is statistically significant in a model that already includes the gaming-related cluster and Feature 36. Due to the non-conclusiveness of the evidence relevant to these features, we will not discuss all of these features in detail, but will briefly mention one that has appeared in prior discussions of gaming. Lessons where a higher proportion of hint sequences told students what to do on the last hint (Feature 61) had marginally significantly more gaming, r2 = 0.14, F(1,19)=3.28, p=0.09. This result is unsurprising, as drilling through hints and typing in a bottom-out hint is one of the easiest and most frequently reported types of [[gaming the system]].		Five other features that were not strongly loaded in the significant factor were marginally associated with gaming. None of these other features is statistically significant in a model that already includes the gaming-related cluster and Feature 36. Due to the non-conclusiveness of the evidence relevant to these features, we will not discuss all of these features in detail, but will briefly mention one that has appeared in prior discussions of gaming. Lessons where a higher proportion of hint sequences told students what to do on the last hint (Feature 61) had marginally significantly more gaming, r2 = 0.14, F(1,19)=3.28, p=0.09. This result is unsurprising, as drilling through hints and typing in a bottom-out hint is one of the easiest and most frequently reported types of [[gaming the system]].

			The off-task behavior model achieved similar predictive power, but was a much less complex model. None of the 6 factors were statistically significantly associated with gaming. Only one of the features was individually statistically significantly associated with off-task behavior: Feature 79, whether or not the lesson was an equation solver unit. Equation solver units had significantly less off-task behavior, just as they had significantly less gaming the system, and the effect was large in magnitude, r2 = 0.55, F(1, 21)=27.29, p<0.001, Bonferroni adjusted p<0.001.

			To put this relationship into better context, we can look at the proportion of time students
			spent off-task in equation-solver lessons as compared to other lessons. On average,
			students spent 4.4% of their time off-task within the equation-solver lessons, much lower
			than is generally seen in intelligent tutor classrooms or, for that matter, in traditional
			classrooms. By contrast, students spent 14.1% of their time off-task within the
			other lessons, a proportion of time-on-task which is much more in line with previous
			observations. The difference in time spent per type of lesson is, as would be expected,
			statistically significant, t(22)=4.48, p<0.001.

	=== Connections to Other PSLC Studies===		=== Connections to Other PSLC Studies===

			This study inspired and led to the upcoming Year 6 study, [[Baker Closing the Loop on Gaming]].

	===Annotated Bibliography===		===Annotated Bibliography===
Line 216:		Line 229:
	Baker, R.S.J.d. (2007b) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.		Baker, R.S.J.d. (2007b) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

	Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (~~under review~~) Educational Software Features that Encourage and Discourage "Gaming the System". ~~Paper under review~~.		Baker, R.S.J.d. (accepted a) Differences Between Intelligent Tutor Lessons, and the Choice to Go Off-Task. To appear in Proceedings of EDM 2009.

			Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (in press a) Educational Software Features that Encourage and Discourage "Gaming the System". To appear in Proceedings of AIED 2009.

	Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]		Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Ryan at 01:29, 16 May 2009

2009-05-16T01:29:22Z

← Older revision		Revision as of 01:29, 16 May 2009
Line 21:		Line 21:
	===Abstract===		===Abstract===

	We are investigating what factors lead students to make specific path choices in the learning space, focusing specifically on the shallow strategy known as [[gaming the system]], and on [[Off-Task Behavior]]. Prior PSLC research has shown that a variety of motivations, attitudes, and affective states are associated with the choice to game the system (Baker et al, 2004; Baker, ~~2007~~; Rodrigo et al, 2007) and the choice of off-task behavior (Baker, ~~2007~~) within intelligent tutoring systems. However, other recent research has found that differences between lessons are on the whole better predictors of gaming than differences between students (Baker, 2007), suggesting that contextual factors associated with a specific tutor unit may be the most important reason why students game the system. Hence, this project is investigating how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy. An extension to this project in 2008-2009 also investigated how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy.		We are investigating what factors lead students to make specific path choices in the learning space, focusing specifically on the shallow strategy known as [[gaming the system]], and on [[Off-Task Behavior]]. Prior PSLC research has shown that a variety of motivations, attitudes, and affective states are associated with the choice to game the system (Baker et al, 2004; Baker, 2007b; Rodrigo et al, 2007) and the choice of off-task behavior (Baker, 2007b) within intelligent tutoring systems. However, other recent research has found that differences between lessons are on the whole better predictors of gaming than differences between students (Baker, 2007), suggesting that contextual factors associated with a specific tutor unit may be the most important reason why students game the system. Hence, this project is investigating how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy. An extension to this project in 2008-2009 also investigated how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy.

	To this end, we have annotated a large proportion of the learning events/transactions in a set of twenty units in the [[Algebra]] LearnLab with descriptions of each unit's content and interface features, using a combination of human coding and educational data mining. We then used data mining to predict gaming and off-task behavior with the content and interface features of the units they occur in. This gives us new insight into why students make specific path choices in the learning space, and explains the prior finding that path choices differ considerably between tutor units.		To this end, we have annotated a large proportion of the learning events/transactions in a set of twenty units in the [[Algebra]] LearnLab with descriptions of each unit's content and interface features, using a combination of human coding and educational data mining. We then used data mining to predict gaming and off-task behavior with the content and interface features of the units they occur in. This gives us new insight into why students make specific path choices in the learning space, and explains the prior finding that path choices differ considerably between tutor units.
Line 55:		Line 55:
	One analytical tool with considerable power to help learning scientists explain the ways students choose to use a learning environment is the [[learning event space]]. In a learning event space, the different paths a student could take are enumerated, and the effects of each path are detailed, both in terms of how the path influences the student’s success within the environment, and the student’s learning. The learning event space model provides a simple way to identify the possible paths and effects; it also provides a concrete way to break down complex research questions into simpler and more concrete questions.		One analytical tool with considerable power to help learning scientists explain the ways students choose to use a learning environment is the [[learning event space]]. In a learning event space, the different paths a student could take are enumerated, and the effects of each path are detailed, both in terms of how the path influences the student’s success within the environment, and the student’s learning. The learning event space model provides a simple way to identify the possible paths and effects; it also provides a concrete way to break down complex research questions into simpler and more concrete questions.

	[[Gaming the system]] is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004). It was earlier hypothesized that gaming stemmed from stable differences in student goals, motivation, and attitudes -- however multiple studies have now suggested that these constructs play only a small role in predicting gaming behavior (Baker et al, 2005; Walonoski & Heffernan, 2006; Baker et al, 2008). By contrast, variation in short-term affective states and the tutor lesson itself appear to play a much larger role in the choice to game (Rodrigo et al, 2007; Baker, ~~2007~~).		[[Gaming the system]] is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004). It was earlier hypothesized that gaming stemmed from stable differences in student goals, motivation, and attitudes -- however multiple studies have now suggested that these constructs play only a small role in predicting gaming behavior (Baker et al, 2005; Walonoski & Heffernan, 2006; Baker et al, 2008). By contrast, variation in short-term affective states and the tutor lesson itself appear to play a much larger role in the choice to game (Rodrigo et al, 2007; Baker, 2007a).

	In this project, we investigate what it is about some tutor lessons that encourages or discourages gaming. This project helps explain why students choose shallow gaming strategies at some learning events and not at others. This contributes to our understanding of learning event spaces, and makes a significant contribution to the PSLC Theoretical Framework, by providing an account for why students choose the shallow learning strategies in many of the learning event space models in the PSLC Theoretical Framework. The study of what lesson features predicted gaming was anticipated to jump-start the process of studying why students choose other shallow learning strategies beyond gaming the system, by providing a methodological template that can be directly applied in future research, as well as initial hypotheses to investigate. It did so, enabling analysis of which lesson features are associated with the choice to go off-task. This study has influenced the upcoming PSLC project [[Baker Closing the Loop on Gaming]].		In this project, we investigate what it is about some tutor lessons that encourages or discourages gaming. This project helps explain why students choose shallow gaming strategies at some learning events and not at others. This contributes to our understanding of learning event spaces, and makes a significant contribution to the PSLC Theoretical Framework, by providing an account for why students choose the shallow learning strategies in many of the learning event space models in the PSLC Theoretical Framework. The study of what lesson features predicted gaming was anticipated to jump-start the process of studying why students choose other shallow learning strategies beyond gaming the system, by providing a methodological template that can be directly applied in future research, as well as initial hypotheses to investigate. It did so, enabling analysis of which lesson features are associated with the choice to go off-task. This study has influenced the upcoming PSLC project [[Baker Closing the Loop on Gaming]].
Line 173:		Line 173:
	===Findings and Explanation===		===Findings and Explanation===

	The text below is taken from (Baker, ~~2007~~; Baker et al, in press a, accepted).		The text below is taken from (Baker, 2007b; Baker et al, in press a, accepted).

	The difference between lessons is a significantly better predictor than the difference between students in determining how much gaming behavior a student will engage in, in a given lesson. Put more simply, knowing which lesson a student is using is a better predictor of how much gaming will occur, than knowing which student it is.		The difference between lessons is a significantly better predictor than the difference between students in determining how much gaming behavior a student will engage in, in a given lesson. Put more simply, knowing which lesson a student is using is a better predictor of how much gaming will occur, than knowing which student it is.
Line 179:		Line 179:
	In the Middle School Tutor, lesson has 35 parameters and achieves an r-squared of 0.55. Student has 240 parameters and achieves an r-squared of 0.16. In the Algebra Tutor, lesson has 21 parameters and achieves an r-squared of 0.18. Student achieves an equal r-squared, but with 58 students; hence, lesson is a statistically better predictor because it achieves equal or significantly better fit with considerably fewer parameters.		In the Middle School Tutor, lesson has 35 parameters and achieves an r-squared of 0.55. Student has 240 parameters and achieves an r-squared of 0.16. In the Algebra Tutor, lesson has 21 parameters and achieves an r-squared of 0.18. Student achieves an equal r-squared, but with 58 students; hence, lesson is a statistically better predictor because it achieves equal or significantly better fit with considerably fewer parameters.

	~~[[Image:algebra-lessons-gaming.jpg]]~~		We empirically grouped the 79 features of the CTLVS1.1 with Principal Component Analysis (PCA). We grouped the 79 features of the CTLVS1 into 6 clusters. We then analyzed whether the correlation between these 6 clusters and the frequency of gaming the system was significant in any case.

	~~This figure shows the prevalence of gaming across units in the [[Algebra]] LearnLab~~

	~~In the [[Algebra]] LearnLab, the degree of gaming does not seem to be obviously related to topic, or order in the curriculum,~~
	~~but appears to be linked to when the unit was developed -- each of the top five most gamed units comes from the original set~~
	~~of units developed for the Algebra I tutor.~~

	We empirically grouped the 79 features of the CTLVS1 with ~~Principle~~ Component Analysis (PCA). We grouped the 79 features of the CTLVS1 into 6 clusters. We then analyzed whether the correlation between these 6 clusters and the frequency of gaming the system was significant in any case.

	Of these 6 clusters, one was statistically significantly associated with the choice to game the system, r2 = 0.29 (e.g. accounting for 29% of the variance in gaming), F(1,19)= 7.84, p=0.01. The cluster loaded strongly on eight features associated with more gaming:		Of these 6 clusters, one was statistically significantly associated with the choice to game the system, r2 = 0.29 (e.g. accounting for 29% of the variance in gaming), F(1,19)= 7.84, p=0.01. The cluster loaded strongly on eight features associated with more gaming:
Line 205:		Line 197:
	Taken individually, two of the constructs in this cluster were significantly (or marginally significantly) associated with gaming. Feature 53-inverse (text in the problem statement not directly related to the problem-solving task) was associated with significantly less gaming, r2 = 0.19, F(1,19) = 4.59, p = 0.04. Feature 40 (when it is not immediately apparent what icons in toolbar mean) was marginally significantly associated with more gaming, r2 = 0.15, F(1, 19)=3.52, p=0.08. The fact that other top features in the cluster were not independently associated with gaming, while the cluster as a whole was fairly strongly associated with gaming, suggests that gaming may occur primarily when more than one of these features are present.		Taken individually, two of the constructs in this cluster were significantly (or marginally significantly) associated with gaming. Feature 53-inverse (text in the problem statement not directly related to the problem-solving task) was associated with significantly less gaming, r2 = 0.19, F(1,19) = 4.59, p = 0.04. Feature 40 (when it is not immediately apparent what icons in toolbar mean) was marginally significantly associated with more gaming, r2 = 0.15, F(1, 19)=3.52, p=0.08. The fact that other top features in the cluster were not independently associated with gaming, while the cluster as a whole was fairly strongly associated with gaming, suggests that gaming may occur primarily when more than one of these features are present.

	~~One feature~~ that ~~was~~ not present in the significant cluster was statistically significantly associated with gaming: Feature 36, where the location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated, r2 = 0.20, F(1,19)=4.97, p=0.04. This feature, like many of those in the gaming-related cluster, represents an unclear or confusing lesson. When the gaming-related cluster and Feature 36 were included in a model together, both remain statistically significant, and the combined model explains 41% of the variance in gaming (e.g. r2 = 0.41).		Two features that were not present in the significant cluster was statistically significantly associated with gaming: Feature 36, where the location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated, r2 = 0.20, F(1,19)=4.97, p=0.04. This feature, like many of those in the gaming-related cluster, represents an unclear or confusing lesson.

			When the gaming-related cluster and Feature 36 were included in a model together, both remain statistically significant, and the combined model explains 41% of the variance in gaming (e.g. r2 = 0.41).

	Five other features that were not strongly loaded in the significant cluster were marginally associated with gaming. None of these other features is statistically significant in a model that already includes the gaming-related cluster and Feature 36. Due to the non-conclusiveness of the evidence relevant to these features, we will not discuss all of these features in detail, but will briefly mention one that has appeared in prior discussions of gaming. Lessons where a higher proportion of hint sequences told students what to do on the last hint (Feature 61) had marginally significantly more gaming, r2 = 0.14, F(1,19)=3.28, p=0.09. This result is unsurprising, as drilling through hints and typing in a bottom-out hint is one of the easiest and most frequently reported types of [[gaming the system]].		Five other features that were not strongly loaded in the significant cluster were marginally associated with gaming. None of these other features is statistically significant in a model that already includes the gaming-related cluster and Feature 36. Due to the non-conclusiveness of the evidence relevant to these features, we will not discuss all of these features in detail, but will briefly mention one that has appeared in prior discussions of gaming. Lessons where a higher proportion of hint sequences told students what to do on the last hint (Feature 61) had marginally significantly more gaming, r2 = 0.14, F(1,19)=3.28, p=0.09. This result is unsurprising, as drilling through hints and typing in a bottom-out hint is one of the easiest and most frequently reported types of [[gaming the system]].

Ryan at 01:27, 16 May 2009

2009-05-16T01:27:40Z

Show changes

Ryan at 01:07, 16 May 2009

2009-05-16T01:07:46Z

← Older revision		Revision as of 01:07, 16 May 2009
Line 21:		Line 21:
	===Abstract===		===Abstract===

	We are investigating what factors lead students to make specific path choices in the learning space, focusing specifically on the shallow strategy known as [[gaming the system]]. Prior research has shown that a variety of motivations, attitudes, and affective states are associated with the choice to game the system (Baker et al, 2004; Baker, 2007; Rodrigo et al, 2007). However, other recent research has found that differences between lessons are on the whole better predictors of gaming than differences between students (Baker, 2007), suggesting that contextual factors associated with a specific tutor unit may be the most important reason why students game the system. Hence, this project is investigating how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy.		We are investigating what factors lead students to make specific path choices in the learning space, focusing specifically on the shallow strategy known as [[gaming the system]], and on [[Off-Task Behavior]]. Prior PSLC research has shown that a variety of motivations, attitudes, and affective states are associated with the choice to game the system (Baker et al, 2004; Baker, 2007; Rodrigo et al, 2007) and the choice of off-task behavior (Baker, 2007) within intelligent tutoring systems. However, other recent research has found that differences between lessons are on the whole better predictors of gaming than differences between students (Baker, 2007), suggesting that contextual factors associated with a specific tutor unit may be the most important reason why students game the system. Hence, this project is investigating how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy.

	To this end, we are in the process of annotating each learning event/transaction in a set of units in the [[Algebra]] LearnLab and the Middle School Cognitive Tutor with descriptions of its content and interface features, using a combination of human coding and machine learning. We are also in the process of adapting a detector of the shallow gaming strategy developed in a subset of the Middle School Cognitive Tutor to the LearnLabs, in order to use the gaming detector to predict exactly which learning events in the data involve shallow gaming strategies. After completing these steps, we will use data mining to connect gaming path choices with the content and interface features of the learning events they occur in. This will give us insight into why students make specific path choices in the learning space, and explain the prior finding that path choices differ considerably between tutor units.		To this end, we are in the process of annotating each learning event/transaction in a set of units in the [[Algebra]] LearnLab and the Middle School Cognitive Tutor with descriptions of its content and interface features, using a combination of human coding and machine learning. We are also in the process of adapting a detector of the shallow gaming strategy developed in a subset of the Middle School Cognitive Tutor to the LearnLabs, in order to use the gaming detector to predict exactly which learning events in the data involve shallow gaming strategies. After completing these steps, we will use data mining to connect gaming path choices with the content and interface features of the learning events they occur in. This will give us insight into why students make specific path choices in the learning space, and explain the prior finding that path choices differ considerably between tutor units.

Ryan at 22:59, 4 December 2008

2008-12-04T22:59:47Z

← Older revision		Revision as of 22:59, 4 December 2008
Line 173:		Line 173:

	[[Image:algebra-lessons-gaming.jpg]]		[[Image:algebra-lessons-gaming.jpg]]

	This figure shows the prevalence of gaming across units in the [[Algebra]] LearnLab		This figure shows the prevalence of gaming across units in the [[Algebra]] LearnLab

Ryan: added results

2008-12-04T22:59:20Z

added results

← Older revision		Revision as of 22:59, 4 December 2008
Line 164:		Line 164:
	These predictions were created by using text replay observations (Baker, Corbett, & Wagner, 2006) to label a representative set of transactions, and then using these labels to create gaming detectors (cf. Baker, Corbett, & Koedinger, 2004; Baker et al, 2008) which can be used to label the remaining transactions.		These predictions were created by using text replay observations (Baker, Corbett, & Wagner, 2006) to label a representative set of transactions, and then using these labels to create gaming detectors (cf. Baker, Corbett, & Koedinger, 2004; Baker et al, 2008) which can be used to label the remaining transactions.

	===Findings===		===Findings and Explanation===

	~~Once all lessons are labeled according to the taxonomy features~~, ~~and all transactions are labeled as gaming or not gaming, we will determine which features of a lesson predict whether students will game more or less. This will occur in Spring 2008~~.		The text below is taken from (Baker et al, under review).

	~~A preliminary finding is that the~~ difference between lessons is a significantly better predictor than the difference between students in determining how much gaming behavior a student will engage in, in a given lesson. Put more simply, knowing which lesson a student is using is a better predictor of how much gaming will occur, than knowing which student it is.		The difference between lessons is a significantly better predictor than the difference between students in determining how much gaming behavior a student will engage in, in a given lesson. Put more simply, knowing which lesson a student is using is a better predictor of how much gaming will occur, than knowing which student it is.

	In the Middle School Tutor, lesson has 35 parameters and achieves an r-squared of 0.55. Student has 240 parameters and achieves an r-squared of 0.16. In the Algebra Tutor, lesson has 21 parameters and achieves an r-squared of 0.18. Student achieves an equal r-squared, but with 58 students; hence, lesson is a statistically better predictor because it achieves equal or significantly better fit with considerably fewer parameters.		In the Middle School Tutor, lesson has 35 parameters and achieves an r-squared of 0.55. Student has 240 parameters and achieves an r-squared of 0.16. In the Algebra Tutor, lesson has 21 parameters and achieves an r-squared of 0.18. Student achieves an equal r-squared, but with 58 students; hence, lesson is a statistically better predictor because it achieves equal or significantly better fit with considerably fewer parameters.
Line 174:		Line 174:
	[[Image:algebra-lessons-gaming.jpg]]		[[Image:algebra-lessons-gaming.jpg]]
	This figure shows the prevalence of gaming across units in the [[Algebra]] LearnLab		This figure shows the prevalence of gaming across units in the [[Algebra]] LearnLab

	~~===Explanation===~~

	In the [[Algebra]] LearnLab, the degree of gaming does not seem to be obviously related to topic, or order in the curriculum,		In the [[Algebra]] LearnLab, the degree of gaming does not seem to be obviously related to topic, or order in the curriculum,
Line 181:		Line 179:
	of units developed for the Algebra I tutor.		of units developed for the Algebra I tutor.

	~~A fuller explanation~~ for this ~~finding~~ will ~~be obtained when~~ the ~~project~~ is ~~completed~~.		We empirically grouped the 79 features of the CTLVS1 with Principle Component Analysis (PCA). We grouped the 79 features of the CTLVS1 into 6 clusters. We then analyzed whether the correlation between these 6 clusters and the frequency of gaming the system was significant in any case.

			Of these 6 clusters, one was statistically significantly associated with the choice to game the system, r2 = 0.29 (e.g. accounting for 29% of the variance in gaming), F(1,19)= 7.84, p=0.01. The cluster loaded strongly on eight features associated with more gaming:
			* 14: The same number being used for multiple constructs
			* 23-inverse-direction: Reading hints does not positively influence performance on future opportunities to use skill
			* 27: Proportion of hints in each hint sequence that refer to abstract principles
			* 40: Not immediately apparent what icons in toolbar mean
			* 53-inverse-direction: Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest
			* 63-inverse-direction: Hints do not give directional feedback such as “try a larger number”
			* 71-inverse-direction: Lack of implementation flaw in hint message, where there is a reference to a non-existent interface component
			* 75: Hint requests that student perform some action

			In general, several of the features in this cluster appear to correspond to a lack of clarity in the presentation of the content or task (23-inverse, 40, 63-inverse), as well as abstractness (27) and ambiguity (14). Curiously, feature 71-inverse (the lack of a specific type of implementation flaw in hint messages, which would make things very unclear) appears to point in the opposite direction – however, this implementation flaw was only common in a single rarely gamed lesson, so this result is probably a statistical artifact.

			Feature 53-inverse appears to represent a different construct – interestingness (or the attempt to increase interestingness). The fact that feature 53 was associated with less gaming whereas more specific interest-increasing features (features 46-52) were not so strongly related may suggest that it is less important exactly how a problem scenario attempts to increase interest, than it is important that the problem scenario has some content in it that is not strictly mathematical.

			Taken individually, two of the constructs in this cluster were significantly (or marginally significantly) associated with gaming. Feature 53-inverse (text in the problem statement not directly related to the problem-solving task) was associated with significantly less gaming, r2 = 0.19, F(1,19) = 4.59, p = 0.04. Feature 40 (when it is not immediately apparent what icons in toolbar mean) was marginally significantly associated with more gaming, r2 = 0.15, F(1, 19)=3.52, p=0.08. The fact that other top features in the cluster were not independently associated with gaming, while the cluster as a whole was fairly strongly associated with gaming, suggests that gaming may occur primarily when more than one of these features are present.

			One feature that was not present in the significant cluster was statistically significantly associated with gaming: Feature 36, where the location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated, r2 = 0.20, F(1,19)=4.97, p=0.04. This feature, like many of those in the gaming-related cluster, represents an unclear or confusing lesson. When the gaming-related cluster and Feature 36 were included in a model together, both remain statistically significant, and the combined model explains 41% of the variance in gaming (e.g. r2 = 0.41).

			Five other features that were not strongly loaded in the significant cluster were marginally associated with gaming. None of these other features is statistically significant in a model that already includes the gaming-related cluster and Feature 36. Due to the non-conclusiveness of the evidence relevant to these features, we will not discuss all of these features in detail, but will briefly mention one that has appeared in prior discussions of gaming. Lessons where a higher proportion of hint sequences told students what to do on the last hint (Feature 61) had marginally significantly more gaming, r2 = 0.14, F(1,19)=3.28, p=0.09. This result is unsurprising, as drilling through hints and typing in a bottom-out hint is one of the easiest and most frequently reported types of [[gaming the system]].

	=== Connections to Other PSLC Studies===		=== Connections to Other PSLC Studies===

Ryan at 22:56, 4 December 2008

2008-12-04T22:56:28Z

Show changes