Theory Wiki - User contributions [en]

Baker - Closing the Loop

2010-08-24T15:57:01Z

Ryan: /* Summary Table */

== Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop” ==
=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker
|-
| '''Other Contributers''' || Milan Desai
|-
| '''Study Start Date''' || Fall, 2010
|-
| '''Study End Date''' || Spring, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===

This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC ([[Baker_Choices_in_LE_Space]]) (Baker et al, 2009), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1 -- full details on this model are given on the page [[Baker_Choices_in_LE_Space]]) was created, and used to study why some tutor lessons are gamed more than others in the Algebra tutor. The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed (CTA 12: Systems of Equations A), and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

=== Background & Significance ===

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as Help Abuse (Aleven & Koedinger, 2001), and Systematic Guessing (Baker et al, 2004). This pair of strategies is referred to as [[Gaming the system]] (Baker et al, 2004). Gaming the system is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004).

Recent work has indicated that a variety of aspects of cognitive tutor lessons are predictive of greater quantities of gaming (Baker et al, 2009). In particular, gaming is predicted by (boldface indicates a particularly strong relationship -- r2>0.15)

* '''Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest'''
* '''Not immediately apparent what icons in toolbar mean'''
* '''The lesson is not an equation-solver unit'''
* '''The location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated'''
* The same number being used for multiple constructs
* Reading hints does not positively influence performance on future opportunities to use skill
* Proportion of hints in each hint sequence that refer to abstract principles
* Hints do not give directional feedback such as “try a larger number”
* Hint requests that student perform some action

Historically, work to address gaming the system has either attempted to eliminate gaming behavior by making it more difficult to game (for instance, by putting delays between hints), or has attempted to detect gaming automatically and respond to it. The first approach appears to lead students to find new gaming strategies (Murray & VanLehn, 2005), and the second approach has led to systems that reduce gaming (Arroyo et al, 2007; Baker et al, 2006; Roll et al, 2007; Walonoski & Heffernan, 2006) and improve learning (Arroyo et al, 2007; Baker et al, 2006), but has typically been time-consuming and difficult to scale.

Within this project, we propose to use the findings from Baker et al (2009), which found tutor lesson features associated with gaming, as design principles for how to develop tutor lessons that students do not choose to game, in a completely unnoticeable way.

This study will also give us the opportunity to make a contribution to another area of research – recent work has suggested that interest-increasing text may reduce the transferability of knowledge (Kaminski et al, 2009). This may form a trade-off, where interest-increasing text reduces gaming (improving learning) but reduces the transferability of the learning. We will investigate this question with conditions that both include interest-increasing text, and lack interest-increasing text, but are identical in all other ways.

A third potential benefit of this study is in elucidating the link between these aspects of tutors and gaming behavior. We hypothesize that the mediating link is via the affective states of boredom and confusion. We will validate this hypothesis by observing student affect in each condition of the study.

=== Glossary ===

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

=== Hypotheses ===

;H1
: Re-designing a tutor lesson to eliminate features shown to be associated with gaming (see above), will result in lower gaming and better learning

;H2
: The tutor lesson features associated with gaming(see above) influence gaming by increasing the incidence of boredom and confusion

;H3
: Adding interest-increasing text to a tutor lesson (one such feature) will reduce gaming and improve learning, but potentially at the cost of lower transfer

=== Independent Variables ===

Three conditions will be compared:

* An unmodified version of a highly-gamed lesson in Cognitive Tutor Algebra (CTA 12: Systems of Equations A)
* A modified version of the same lesson
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature
* A second modified version of the same lesson
** Adding interest-increasing extraneous text to the scenario
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature

=== Dependent Variables ===

* Incidence of gaming behavior (measured via quantitative field observations -- cf. Baker et al, 2004)
* Incidence of boredom and confusion (measured via quantitative field observations -- cf. Rodrigo et al, 2007)
* Learning of domain skills and concepts (measured pre-post)
* Transfer and preparation for future learning (measured at post-test)

=== Planned Experiments ===

A comparison of the three conditions will be conducted in an in-vivo school study.
We will control for time, and assign students randomly to conditions.

=== Explanation ===

=== Further Information ===

=== Connections ===
[[Baker_Choices_in_LE_Space]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58

Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.

Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A.,
Mahadevan, S., and Woolf. B.P. (2007) Repairing Disengagement with Non-
Invasive Interventions. Proceedings of the 13h International Conference on
Artificial Intelligence in Education, 195-202.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S.E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.

Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.

Kaminski, J.A., Sloutsky, V.M., Heckler, A. (2009) Transfer of Mathematical Knowledge: The Portability of Generic Instantiations. Child Development, 3 (3), 151-155.

Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While
Providing Proactive Help. Proc. of the International Conference on Artificial Intelligence in
Education, 887-889.

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]

Roll, I., Aleven, V., McLaren, B.M., and Koedinger, K.R. (2007) Can help seeking be
tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of
the 13th International Conference on Artificial Intelligence in Education, 203-210.

Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.

Walonoski, J.A., Heffernan, N.T. (2006) Prevention of Off-Task Gaming Behavior in
Intelligent Tutoring Systems. Proceedings of the 8th International Conference on
Intelligent Tutoring Systems, 722-724.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2010-08-24T15:56:20Z

Ryan: /* Research Plan */

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, MathTutor, ScienceAssistments
|-
| '''Number of Students''' || 78 so far; total TBD
|-
| '''Total Participant Hours''' || 444 so far; total TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, the Chemistry Virtual Lab, MathTutor, and Science ASSISTments. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data -- software development completed, as of Aug 2010 synchronization verification in progress

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7) -- as of Aug 2010 first data set collected, other data collection in progress

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al, 2010)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010, data will be collected in the Algebra, Geometry, Chemistry, MathTutor, and Science ASSISTments.

=== Explanation ===
=== Further Information ===
=== Connections ===

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2010-08-24T15:55:43Z

Ryan: /* Connections */

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, MathTutor, ScienceAssistments
|-
| '''Number of Students''' || 78 so far; total TBD
|-
| '''Total Participant Hours''' || 444 so far; total TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, the Chemistry Virtual Lab, MathTutor, and Science ASSISTments. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data -- software development completed, synchronization verification in progress

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7) -- first data set collected, other data collection in progress

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al, 2010)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010, data will be collected in the Algebra, Geometry, Chemistry, MathTutor, and Science ASSISTments.

=== Explanation ===
=== Further Information ===
=== Connections ===

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2010-08-24T15:55:35Z

Ryan: /* Planned Studites */

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, MathTutor, ScienceAssistments
|-
| '''Number of Students''' || 78 so far; total TBD
|-
| '''Total Participant Hours''' || 444 so far; total TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, the Chemistry Virtual Lab, MathTutor, and Science ASSISTments. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data -- software development completed, synchronization verification in progress

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7) -- first data set collected, other data collection in progress

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al, 2010)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010, data will be collected in the Algebra, Geometry, Chemistry, MathTutor, and Science ASSISTments.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2010-08-24T15:55:08Z

Ryan: /* Affective States and M&M Behaviors to be Modeled */

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, MathTutor, ScienceAssistments
|-
| '''Number of Students''' || 78 so far; total TBD
|-
| '''Total Participant Hours''' || 444 so far; total TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, the Chemistry Virtual Lab, MathTutor, and Science ASSISTments. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data -- software development completed, synchronization verification in progress

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7) -- first data set collected, other data collection in progress

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al, 2010)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010 and 2011, data will be collected in the Algebra, Geometry, Chemistry, and Chinese LearnLabs.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2010-08-24T15:54:15Z

Ryan: /* Research Plan */

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, MathTutor, ScienceAssistments
|-
| '''Number of Students''' || 78 so far; total TBD
|-
| '''Total Participant Hours''' || 444 so far; total TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, the Chemistry Virtual Lab, MathTutor, and Science ASSISTments. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data -- software development completed, synchronization verification in progress

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7) -- first data set collected, other data collection in progress

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al under, review)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Confusion (D'Mello et al, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010 and 2011, data will be collected in the Algebra, Geometry, Chemistry, and Chinese LearnLabs.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2010-08-24T15:53:02Z

Ryan: /* Research Process */

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, MathTutor, ScienceAssistments
|-
| '''Number of Students''' || 78 so far; total TBD
|-
| '''Total Participant Hours''' || 444 so far; total TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, the Chemistry Virtual Lab, MathTutor, and Science ASSISTments. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data, and questionnaire prompting (months 1-3) (in coordination with [[Nokes - Questionnaires]])

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7)

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)
* Randomized infrequent polling of student affect, motivation in popup windows
(“Which of these best describes how you’re feeling? [frustrated] [bored] [etc.]”) (in coordination with [[Nokes - Questionnaires]])

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al under, review)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Confusion (D'Mello et al, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010 and 2011, data will be collected in the Algebra, Geometry, Chemistry, and Chinese LearnLabs.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2010-08-24T15:52:20Z

Ryan: /* Study 1 */

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, MathTutor, ScienceAssistments
|-
| '''Number of Students''' || 78 so far; total TBD
|-
| '''Total Participant Hours''' || 444 so far; total TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, Chinese/FaCT, and the Chemistry Virtual Lab. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data, and questionnaire prompting (months 1-3) (in coordination with [[Nokes - Questionnaires]])

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7)

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)
* Randomized infrequent polling of student affect, motivation in popup windows
(“Which of these best describes how you’re feeling? [frustrated] [bored] [etc.]”) (in coordination with [[Nokes - Questionnaires]])

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al under, review)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Confusion (D'Mello et al, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010 and 2011, data will be collected in the Algebra, Geometry, Chemistry, and Chinese LearnLabs.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2010-08-24T15:51:51Z

Ryan: /* Study 1 */

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, Chinese
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, Chinese/FaCT, and the Chemistry Virtual Lab. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data, and questionnaire prompting (months 1-3) (in coordination with [[Nokes - Questionnaires]])

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7)

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)
* Randomized infrequent polling of student affect, motivation in popup windows
(“Which of these best describes how you’re feeling? [frustrated] [bored] [etc.]”) (in coordination with [[Nokes - Questionnaires]])

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al under, review)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Confusion (D'Mello et al, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010 and 2011, data will be collected in the Algebra, Geometry, Chemistry, and Chinese LearnLabs.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Math Game Elements

2010-01-07T20:13:41Z

Ryan: /* Study 1 */

== Improving student affect through adding game elements to mathematics LearnLabs ==
=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Vincent Aleven, Ryan Baker
|-
| '''Other Contributers''' || Adriana de Carvalho (Research Associate, CMU HCII); Owen Durni, and Matt Morrill (Undergraduate research assistant, and former PSLC summer intern); Michael Ringenberg, Shawn Snyder, Martin van Velsen (Research Programmers, CMU HCII); BJ Nartker(Artist, independent contractor)

|-
| '''Study Start Date''' || Fall, 2010
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Middle-School Mathematics
|-
| '''Number of Students''' ||
|-
| '''Total Participant Hours''' ||
|-
| '''DataShop''' ||
|}

=== Abstract ===
There is much evidence to believe that games are fun. Can we incorporate some of the features that make games fun into intelligent tutors, in a way that improves motivation, generates positive affect, and improves the robustness of student learning? Specifically, what happens if we take game elements known to be effective such as fantasy, competition, and trivial choice, and embed into tutors already known to promote learning, using principles in PSLC theoretical framework? In the current project (which started in Year 5 of the PSLC), we are investigating the effect of adding game elements to an existing set of fractions tutors developed by Martina Rau (Rau, Aleven, & Rummel, in press) for a different PSLC project. The game elements comprise a fantasy soccer game, where success in the soccer game depends on learning progress in the factions tutors.

=== Background & Significance ===
Games and tutors appear to have complementary strengths. The current project is an attempt to determine if we can develop learning environments that leverage the strength of each type of learning environment, creating learning software that is as motivationally effective as games but promotes robust learning as well as intelligent tutors. We investigate one particular way of integrating game elements and learning content, building a game around an intelligent tutor engine.

How best to integrate learning content and game elements has been the subject of much theorizing, with some authors arguing that optimal learning requires that the learning content and game world be mutually dependent (Lepper & Malone, 1987), and others arguing that the learning should be embedded in the game’s core mechanic (Habgood, 2007). However, such theories ignore the real-world success (at least in anecdotal reports from teachers and students) of environments that feature a much looser integration between learning and motivational embellishments (e.g., FirstInMath). Given that a loose coupling between game elements and learning activities is far easier to implement (since it avoids the difficult problem of embedding math problems in a storyline or game context, hard to do especially if the learning content is to be adapted to individual students’ learning results), it is reasonable to investigate this option first. It may well be that as long as the game features are “cool,” the degree of integration is not really a strong factor. As mentioned, the success of for example the motivational embellishments in FirstInMath certainly suggest so.

=== Glossary ===

[[Metacognition and Motivation]]

=== Hypotheses ===

;H1
: A tutor with game features will lead to equal robust-learning outcomes as an unmodified tutor covering the same material

;H2
: A tutor with game features will lead to significantly better affect (e.g. less boredom and frustration; more delight) than an unmodified tutor covering the same material

;H3
: Students will report higher liking of a tutor with gaming features than an unmodified tutor covering the same material

;H4
: Students will choose to play the tutor with game features for longer than an unmodified tutor covering the same material, given the option to choose other activities.

=== Independent Variables ===
We are investigating the effect of loosely coupling the fractions tutors to a parallel fantasy world. This fantasy world was designed in a participatory design process involving 6th graders (the target population). The main theme (soccer) was chosen because it will appeal to the taste of both girls and boys.

In the fantasy world, a soccer game takes place on a tropical beach between a motley crew of friendly animal characters and a group of invading pirates. This unfriendly bunch is intent on taking over the beach, a disaster that can be averted only by beating the pirates at the soccer game. The student regularly switches between the tutors and the fantasy world (in a manner controlled largely by the system). During each visit to the fantasy world, the student selects a game move (e.g., a deep pass, or less risky dribble). The outcome of that move will not be known until the student’s next visit to the fantasy world, and the probability of success of that move depends on the student’s performance on the intervening fractions problems. (It may be clear that this coupling between game elements and learning is loose: different learning content could be slotted in without changing the game.)

 <center>[[Image:Amigos.jpg]] An image from the "pirate soccer" fantasy world</center> 

=== Dependent Variables ===
We will evaluate whether this game leads to increased learning, compared to a standard tutor without the game elements, and whether any improvement in learning we may observe is mediated by increased motivation and more positive affect. Affect will be assessed by means of field observations, motivation (interest, self-efficacy) by means of questionnaires. We will evaluate both whether the game leads to greater persistence (when student have the option to do something else), and whether it leads to greater learning even when the set of learning activities (but not time) is held constant (which if true would mean that the game elements lead to deeper processing during the same set of learning activities).

=== Planned Experiments ===
We will conduct two in-vivo experiments comparing the tutor with game features against the regular tutor.

The first experiment, to be conducted in LearnLab schools in Fall 2009, will control for time. Students will be randomly selected to use either the game or the unmodified tutor (for equity, all students will receive access to the game over the web after the study). Motivation and liking will be assessed by pre-test and post-test questionnaires, and affect will be assessed by quantitative field observations during usage. Robust learning will be measured by pre-test and post-test.

The second experiment, to be conducted in LearnLab schools in Spring 2010, will allow time to vary . Students will be randomly selected to use either the game or the unmodified tutor (for equity, all students will receive access to the game over the web after the study). Students will be required to use the condition for one class period, and then in two subsequent class periods will be given the choice to switch conditions or use an alternate piece of educational software covering the same material. Motivation will be assessed by students' time allocation once they are given the option of switching tasks.

=== Explanation ===
=== Further Information ===
==== Connections ====
==== Annotated Bibliography ====
==== References ====
==== Future Plans ====

Math Game Elements

2010-01-07T20:12:37Z

Ryan:

== Improving student affect through adding game elements to mathematics LearnLabs ==
=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Vincent Aleven, Ryan Baker
|-
| '''Other Contributers''' || Adriana de Carvalho (Research Associate, CMU HCII); Owen Durni, and Matt Morrill (Undergraduate research assistant, and former PSLC summer intern); BJ Nartker(Artist, independent contractor)

|-
| '''Study Start Date''' || Fall, 2010
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Middle-School Mathematics
|-
| '''Number of Students''' ||
|-
| '''Total Participant Hours''' ||
|-
| '''DataShop''' ||
|}

=== Abstract ===
There is much evidence to believe that games are fun. Can we incorporate some of the features that make games fun into intelligent tutors, in a way that improves motivation, generates positive affect, and improves the robustness of student learning? Specifically, what happens if we take game elements known to be effective such as fantasy, competition, and trivial choice, and embed into tutors already known to promote learning, using principles in PSLC theoretical framework? In the current project (which started in Year 5 of the PSLC), we are investigating the effect of adding game elements to an existing set of fractions tutors developed by Martina Rau (Rau, Aleven, & Rummel, in press) for a different PSLC project. The game elements comprise a fantasy soccer game, where success in the soccer game depends on learning progress in the factions tutors.

=== Background & Significance ===
Games and tutors appear to have complementary strengths. The current project is an attempt to determine if we can develop learning environments that leverage the strength of each type of learning environment, creating learning software that is as motivationally effective as games but promotes robust learning as well as intelligent tutors. We investigate one particular way of integrating game elements and learning content, building a game around an intelligent tutor engine.

How best to integrate learning content and game elements has been the subject of much theorizing, with some authors arguing that optimal learning requires that the learning content and game world be mutually dependent (Lepper & Malone, 1987), and others arguing that the learning should be embedded in the game’s core mechanic (Habgood, 2007). However, such theories ignore the real-world success (at least in anecdotal reports from teachers and students) of environments that feature a much looser integration between learning and motivational embellishments (e.g., FirstInMath). Given that a loose coupling between game elements and learning activities is far easier to implement (since it avoids the difficult problem of embedding math problems in a storyline or game context, hard to do especially if the learning content is to be adapted to individual students’ learning results), it is reasonable to investigate this option first. It may well be that as long as the game features are “cool,” the degree of integration is not really a strong factor. As mentioned, the success of for example the motivational embellishments in FirstInMath certainly suggest so.

=== Glossary ===

[[Metacognition and Motivation]]

=== Hypotheses ===

;H1
: A tutor with game features will lead to equal robust-learning outcomes as an unmodified tutor covering the same material

;H2
: A tutor with game features will lead to significantly better affect (e.g. less boredom and frustration; more delight) than an unmodified tutor covering the same material

;H3
: Students will report higher liking of a tutor with gaming features than an unmodified tutor covering the same material

;H4
: Students will choose to play the tutor with game features for longer than an unmodified tutor covering the same material, given the option to choose other activities.

=== Independent Variables ===
We are investigating the effect of loosely coupling the fractions tutors to a parallel fantasy world. This fantasy world was designed in a participatory design process involving 6th graders (the target population). The main theme (soccer) was chosen because it will appeal to the taste of both girls and boys.

In the fantasy world, a soccer game takes place on a tropical beach between a motley crew of friendly animal characters and a group of invading pirates. This unfriendly bunch is intent on taking over the beach, a disaster that can be averted only by beating the pirates at the soccer game. The student regularly switches between the tutors and the fantasy world (in a manner controlled largely by the system). During each visit to the fantasy world, the student selects a game move (e.g., a deep pass, or less risky dribble). The outcome of that move will not be known until the student’s next visit to the fantasy world, and the probability of success of that move depends on the student’s performance on the intervening fractions problems. (It may be clear that this coupling between game elements and learning is loose: different learning content could be slotted in without changing the game.)

 <center>[[Image:Amigos.jpg]] An image from the "pirate soccer" fantasy world</center> 

=== Dependent Variables ===
We will evaluate whether this game leads to increased learning, compared to a standard tutor without the game elements, and whether any improvement in learning we may observe is mediated by increased motivation and more positive affect. Affect will be assessed by means of field observations, motivation (interest, self-efficacy) by means of questionnaires. We will evaluate both whether the game leads to greater persistence (when student have the option to do something else), and whether it leads to greater learning even when the set of learning activities (but not time) is held constant (which if true would mean that the game elements lead to deeper processing during the same set of learning activities).

=== Planned Experiments ===
We will conduct two in-vivo experiments comparing the tutor with game features against the regular tutor.

The first experiment, to be conducted in LearnLab schools in Fall 2009, will control for time. Students will be randomly selected to use either the game or the unmodified tutor (for equity, all students will receive access to the game over the web after the study). Motivation and liking will be assessed by pre-test and post-test questionnaires, and affect will be assessed by quantitative field observations during usage. Robust learning will be measured by pre-test and post-test.

The second experiment, to be conducted in LearnLab schools in Spring 2010, will allow time to vary . Students will be randomly selected to use either the game or the unmodified tutor (for equity, all students will receive access to the game over the web after the study). Students will be required to use the condition for one class period, and then in two subsequent class periods will be given the choice to switch conditions or use an alternate piece of educational software covering the same material. Motivation will be assessed by students' time allocation once they are given the option of switching tasks.

=== Explanation ===
=== Further Information ===
==== Connections ====
==== Annotated Bibliography ====
==== References ====
==== Future Plans ====

DataShop Feature Wish List

2009-12-07T18:50:43Z

Ryan: /* Prioritized Features */

Below are two lists of features. The features that we have prioritized and decided to implement are in the first, ordered list. The features that the DataShop team and community are discussing are in an unordered list on the page [[Collected User Requests]]. Click on a feature to get more information about it, such as a description, rationale for building it, and its status.

'''You can help!'''

If you think a feature is important, vote for it by putting your name to the right of the feature. Discuss the feature on the comments section of that feature's page. We'll use these votes and the dialogue that develops to prioritize features.

Don't see a feature on the prioritized list? There's a good chance it's on the '''[[Collected User Requests]]''' page. You can add feature ideas there and discuss the existing ones. Include your comment, name, and date to vote on feature ideas there.

'''Tip:''' Easily sign your username and the current date/time by inserting four tildes (<nowiki>~~~~</nowiki>); insert just your username with three tildes.

''See features we are building now, [[DataShop 4.x Features]], and ones we have built, [[DataShop 3.x Features]].''

== Prioritized Features ==

# [[Web Services]] (add custom fields to transactions) — Vote: Ryan Baker(1), John Stamper(1)
# [[Adding Custom Fields through Web Application]] -- Vote: Ryan Baker (2)
# [[Error Bars]] — Vote: Ken Koedinger(1)
# [[Metrics]] — Vote: John Stamper (2), Ryan Baker (3)
# [[Push Button Import]] — Carnegie Learning, John Stamper
# [[KC Model in Transaction Export]] — Vote: Vincent Aleven(2)
# [[Student Filter Dialog]]
# [[Milliseconds]]
# [[LFA on Sample]] — Vote: Ken Koedinger(3)
# [[Place for General Papers]]
# [[Performance Metrics]]
# [[KC Model Sort]]
# [[Ability to display step-custom-fields in graphs]]
# [[Scalability]] -- Vote: Ryan Baker (4)
# [[Terms of Use]] — Steve Ritter

== Unordered Features ==
We have a long list of feature requests that have not been prioritized. Please see the
'''[[Collected User Requests]]'''.

== How to Request a Feature ==
* [[Write a User Story]]
* [[Create a Feature Page]]
* Add Link to Feature on [[Collected User Requests]] page.
 
----
See completed [[DataShop 3.x Features]] 
See on-going [[DataShop 4.x Features]] 
See unordered [[Collected User Requests]]
[[Category:Protected]]
[[Category:DataShop]]

DataShop Feature Wish List

2009-12-07T17:13:37Z

Ryan: /* Prioritized Features */

Below are two lists of features. The features that we have prioritized and decided to implement are in the first, ordered list. The features that the DataShop team and community are discussing are in an unordered list on the page [[Collected User Requests]]. Click on a feature to get more information about it, such as a description, rationale for building it, and its status.

'''You can help!'''

If you think a feature is important, vote for it by putting your name to the right of the feature. Discuss the feature on the comments section of that feature's page. We'll use these votes and the dialogue that develops to prioritize features.

Don't see a feature on the prioritized list? There's a good chance it's on the '''[[Collected User Requests]]''' page. You can add feature ideas there and discuss the existing ones. Include your comment, name, and date to vote on feature ideas there.

'''Tip:''' Easily sign your username and the current date/time by inserting four tildes (<nowiki>~~~~</nowiki>); insert just your username with three tildes.

''See features we are building now, [[DataShop 4.x Features]], and ones we have built, [[DataShop 3.x Features]].''

== Prioritized Features ==

# [[Web Services]] (add custom fields to transactions) — Vote: Ryan Baker(1), John Stamper(1)
# [[Adding Custom Fields through Web Application]] -- Vote: Ryan Baker (1)
# [[Error Bars]] — Vote: Ken Koedinger(1)
# [[Metrics]] — Vote: John Stamper (2), Ryan Baker (1)
# [[Push Button Import]] — Carnegie Learning, John Stamper
# [[KC Model in Transaction Export]] — Vote: Vincent Aleven(2)
# [[Student Filter Dialog]]
# [[Milliseconds]]
# [[LFA on Sample]] — Vote: Ken Koedinger(3)
# [[Place for General Papers]]
# [[Performance Metrics]]
# [[KC Model Sort]]
# [[Ability to display step-custom-fields in graphs]]
# [[Scalability]]
# [[Terms of Use]] — Steve Ritter

== Unordered Features ==
We have a long list of feature requests that have not been prioritized. Please see the
'''[[Collected User Requests]]'''.

== How to Request a Feature ==
* [[Write a User Story]]
* [[Create a Feature Page]]
* Add Link to Feature on [[Collected User Requests]] page.
 
----
See completed [[DataShop 3.x Features]] 
See on-going [[DataShop 4.x Features]] 
See unordered [[Collected User Requests]]
[[Category:Protected]]
[[Category:DataShop]]

Metrics

2009-12-07T17:12:14Z

Ryan: /* Notes/Comments */

'''Status: Requirements Document in Progress'''

== User Story ==

As an administrator of DataShop, I want to see dataset metrics that are updated at a regular interval so that we can easily put tables on posters for important meetings such as NSF Site Visits, Advisory Board Visits, etc.

== Notes/Comments ==

* Eventually we'd like to graph this data over time. -- I really like this idea! [Ryan Baker]
 

----

See complete [[DataShop Feature Wish List]].

[[Category:Protected]] [[Category:DataShop]]

DataShop Feature Wish List

2009-12-07T17:11:35Z

Ryan: /* Prioritized Features */

Below are two lists of features. The features that we have prioritized and decided to implement are in the first, ordered list. The features that the DataShop team and community are discussing are in an unordered list on the page [[Collected User Requests]]. Click on a feature to get more information about it, such as a description, rationale for building it, and its status.

'''You can help!'''

If you think a feature is important, vote for it by putting your name to the right of the feature. Discuss the feature on the comments section of that feature's page. We'll use these votes and the dialogue that develops to prioritize features.

Don't see a feature on the prioritized list? There's a good chance it's on the '''[[Collected User Requests]]''' page. You can add feature ideas there and discuss the existing ones. Include your comment, name, and date to vote on feature ideas there.

'''Tip:''' Easily sign your username and the current date/time by inserting four tildes (<nowiki>~~~~</nowiki>); insert just your username with three tildes.

''See features we are building now, [[DataShop 4.x Features]], and ones we have built, [[DataShop 3.x Features]].''

== Prioritized Features ==

# [[Web Services]] (add custom fields to transactions) — Vote: Ryan Baker(1), John Stamper(1)
# [[Adding Custom Fields through Web Application]] -- Vote: Ryan Baker (1)
# [[Error Bars]] — Vote: Ken Koedinger(1)
# [[Metrics]] — Vote: John Stamper (2)
# [[Push Button Import]] — Carnegie Learning, John Stamper
# [[KC Model in Transaction Export]] — Vote: Vincent Aleven(2)
# [[Student Filter Dialog]]
# [[Milliseconds]]
# [[LFA on Sample]] — Vote: Ken Koedinger(3)
# [[Place for General Papers]]
# [[Performance Metrics]]
# [[KC Model Sort]]
# [[Ability to display step-custom-fields in graphs]]
# [[Scalability]]
# [[Terms of Use]] — Steve Ritter

== Unordered Features ==
We have a long list of feature requests that have not been prioritized. Please see the
'''[[Collected User Requests]]'''.

== How to Request a Feature ==
* [[Write a User Story]]
* [[Create a Feature Page]]
* Add Link to Feature on [[Collected User Requests]] page.
 
----
See completed [[DataShop 3.x Features]] 
See on-going [[DataShop 4.x Features]] 
See unordered [[Collected User Requests]]
[[Category:Protected]]
[[Category:DataShop]]

Computational Modeling and Data Mining

2009-12-07T16:24:02Z

Ryan: /* References */

==Introduction==
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.

The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.

== Developing Better Cognitive Models of ''Domain-Specific Content''==
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?

One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.

==Developing Models of ''Domain-General'' Learning and Motivational Processes==
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007).

A key Year 5 effort will extend the [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Analysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers.

Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.

==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance.

We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.

== [[CMDM Meetings]] ==

== Descendants ==

To create a new project page, enclose your project name in a double set of brackets. Details for a project format may be [[ Project_Page_Template_and_Creation_Instructions | found here.]]

*[[Gordon - Temporal learning for EDM]]
*[[Koedinger - Discovery of Domain-Specific Cognitive Models]]
*[[Koedinger - Toward a model of accelerated future learning]]
*[[Baker - Building Generalizable Fine-grained Detectors]]
*[[Chi - Induction of Adaptive Pedagogical Tutorial Tactics]]
*[[Baker - Closing the Loop]]
*[[Pavlik - Generalizing the Assistance Formula]]
*[[Mayer_and_McLaren_-_Social_Intelligence_And_Computer_Tutors | McLaren and Mayer - Social Intelligence and Learning from "polite" tutors]]
*[[Application of SimStudent for Error Analysis | Matsuda - Application of SimStudent for Error Analysis]]

== References ==
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.
* Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.
* Baker, R.S.J.d., Corbett, A.T., Aleven, V. (2008) More Accurate Student Modeling Through Contextual Estimation of Slip and Guess Probabilities in Bayesian Knowledge Tracing. Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 406-415
* Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (2008) Developing a Generalizable Detector of When Students Game the System. User Modeling and User-Adapted Interaction, 18, 3, 287-314.
* Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., Koedinger, K. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224.
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.
* McLaren, B.M., Lim, S., & Koedinger, K.R. (2008). When and How Often Should Worked Examples be Given to Students? New Results and a Summary of the Current State of Research. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 2176-2181). Austin, TX: Cognitive Science Society.
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190
* Rodrigo, M.M.T., Baker, R.S.J.d., d'Mello, S., Gonzalez, M.C.T., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sugay, J.O., Tep, S., Viehland, N.J.B. (2008) Comparing Learners' Affect While Using an Intelligent Tutoring Systems and a Simulation Problem Solving Game. Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 40-49.
* Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152.
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.

Computational Modeling and Data Mining

2009-12-07T15:58:38Z

Ryan:

==Introduction==
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.

The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.

== Developing Better Cognitive Models of ''Domain-Specific Content''==
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?

One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.

==Developing Models of ''Domain-General'' Learning and Motivational Processes==
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007).

A key Year 5 effort will extend the [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Analysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers.

Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.

==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance.

We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.

== [[CMDM Meetings]] ==

== Descendants ==

To create a new project page, enclose your project name in a double set of brackets. Details for a project format may be [[ Project_Page_Template_and_Creation_Instructions | found here.]]

*[[Gordon - Temporal learning for EDM]]
*[[Koedinger - Discovery of Domain-Specific Cognitive Models]]
*[[Koedinger - Toward a model of accelerated future learning]]
*[[Baker - Building Generalizable Fine-grained Detectors]]
*[[Chi - Induction of Adaptive Pedagogical Tutorial Tactics]]
*[[Baker - Closing the Loop]]
*[[Pavlik - Generalizing the Assistance Formula]]
*[[Mayer_and_McLaren_-_Social_Intelligence_And_Computer_Tutors | McLaren and Mayer - Social Intelligence and Learning from "polite" tutors]]
*[[Application of SimStudent for Error Analysis | Matsuda - Application of SimStudent for Error Analysis]]

== References ==
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.
* Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.
* Baker, R.S.J.d., Corbett, A.T., Aleven, V. (2008) More Accurate Student Modeling Through Contextual Estimation of Slip and Guess Probabilities in Bayesian Knowledge Tracing. Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 406-415
* Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (2008) Developing a Generalizable Detector of When Students Game the System. User Modeling and User-Adapted Interaction, 18, 3, 287-314.
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.
* McLaren, B.M., Lim, S., & Koedinger, K.R. (2008). When and How Often Should Worked Examples be Given to Students? New Results and a Summary of the Current State of Research. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 2176-2181). Austin, TX: Cognitive Science Society.
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190
* Rodrigo, M.M.T., Baker, R.S.J.d., d'Mello, S., Gonzalez, M.C.T., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sugay, J.O., Tep, S., Viehland, N.J.B. (2008) Comparing Learners' Affect While Using an Intelligent Tutoring Systems and a Simulation Problem Solving Game. Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 40-49.
* Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152.
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.

Computational Modeling and Data Mining

2009-12-07T15:56:47Z

Ryan:

==Introduction==
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.

The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.

== Developing Better Cognitive Models of ''Domain-Specific Content''==
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?

One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.

==Developing Models of ''Domain-General'' Learning and Motivational Processes==
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007).

A key Year 5 effort will extend the [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Analysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers.

Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.

==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance.

We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.

== [[CMDM Meetings]] ==

== Descendants ==

To create a new project page, enclose your project name in a double set of brackets. Details for a project format may be [[ Project_Page_Template_and_Creation_Instructions | found here.]]

*[[Gordon - Temporal learning for EDM]]
*[[Koedinger - Discovery of Domain-Specific Cognitive Models]]
*[[Koedinger - Toward a model of accelerated future learning]]
*[[Baker - Building Generalizable Fine-grained Detectors]]
*[[Chi - Induction of Adaptive Pedagogical Tutorial Tactics]]
*[[Baker - Closing the Loop]]
*[[Pavlik - Generalizing the Assistance Formula]]
*[[Mayer_and_McLaren_-_Social_Intelligence_And_Computer_Tutors | McLaren and Mayer - Social Intelligence and Learning from "polite" tutors]]
*[[Application of SimStudent for Error Analysis | Matsuda - Application of SimStudent for Error Analysis]]

== References ==
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.
* Baker, R.S.J.d., Corbett, A.T., Aleven, V. (2008) More Accurate Student Modeling Through Contextual Estimation of Slip and Guess Probabilities in Bayesian Knowledge Tracing. Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 406-415
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.
* McLaren, B.M., Lim, S., & Koedinger, K.R. (2008). When and How Often Should Worked Examples be Given to Students? New Results and a Summary of the Current State of Research. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 2176-2181). Austin, TX: Cognitive Science Society.
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.

Baker - Closing the Loop

2009-12-05T21:38:38Z

Ryan: /* Planned Experiments */

== Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop” ==
=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker
|-
| '''Other Contributers''' ||
|-
| '''Study Start Date''' || Spring, 2010
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===

This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC ([[Baker_Choices_in_LE_Space]]) (Baker et al, 2009), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1 -- full details on this model are given on the page [[Baker_Choices_in_LE_Space]]) was created, and used to study why some tutor lessons are gamed more than others in the Algebra tutor. The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed (CTA 12: Systems of Equations A), and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

=== Background & Significance ===

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as Help Abuse (Aleven & Koedinger, 2001), and Systematic Guessing (Baker et al, 2004). This pair of strategies is referred to as [[Gaming the system]] (Baker et al, 2004). Gaming the system is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004).

Recent work has indicated that a variety of aspects of cognitive tutor lessons are predictive of greater quantities of gaming (Baker et al, 2009). In particular, gaming is predicted by (boldface indicates a particularly strong relationship -- r2>0.15)

* '''Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest'''
* '''Not immediately apparent what icons in toolbar mean'''
* '''The lesson is not an equation-solver unit'''
* '''The location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated'''
* The same number being used for multiple constructs
* Reading hints does not positively influence performance on future opportunities to use skill
* Proportion of hints in each hint sequence that refer to abstract principles
* Hints do not give directional feedback such as “try a larger number”
* Hint requests that student perform some action

Historically, work to address gaming the system has either attempted to eliminate gaming behavior by making it more difficult to game (for instance, by putting delays between hints), or has attempted to detect gaming automatically and respond to it. The first approach appears to lead students to find new gaming strategies (Murray & VanLehn, 2005), and the second approach has led to systems that reduce gaming (Arroyo et al, 2007; Baker et al, 2006; Roll et al, 2007; Walonoski & Heffernan, 2006) and improve learning (Arroyo et al, 2007; Baker et al, 2006), but has typically been time-consuming and difficult to scale.

Within this project, we propose to use the findings from Baker et al (2009), which found tutor lesson features associated with gaming, as design principles for how to develop tutor lessons that students do not choose to game, in a completely unnoticeable way.

This study will also give us the opportunity to make a contribution to another area of research – recent work has suggested that interest-increasing text may reduce the transferability of knowledge (Kaminski et al, 2009). This may form a trade-off, where interest-increasing text reduces gaming (improving learning) but reduces the transferability of the learning. We will investigate this question with conditions that both include interest-increasing text, and lack interest-increasing text, but are identical in all other ways.

A third potential benefit of this study is in elucidating the link between these aspects of tutors and gaming behavior. We hypothesize that the mediating link is via the affective states of boredom and confusion. We will validate this hypothesis by observing student affect in each condition of the study.

=== Glossary ===

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

=== Hypotheses ===

;H1
: Re-designing a tutor lesson to eliminate features shown to be associated with gaming (see above), will result in lower gaming and better learning

;H2
: The tutor lesson features associated with gaming(see above) influence gaming by increasing the incidence of boredom and confusion

;H3
: Adding interest-increasing text to a tutor lesson (one such feature) will reduce gaming and improve learning, but potentially at the cost of lower transfer

=== Independent Variables ===

Three conditions will be compared:

* An unmodified version of a highly-gamed lesson in Cognitive Tutor Algebra (CTA 12: Systems of Equations A)
* A modified version of the same lesson
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature
* A second modified version of the same lesson
** Adding interest-increasing extraneous text to the scenario
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature

=== Dependent Variables ===

* Incidence of gaming behavior (measured via quantitative field observations -- cf. Baker et al, 2004)
* Incidence of boredom and confusion (measured via quantitative field observations -- cf. Rodrigo et al, 2007)
* Learning of domain skills and concepts (measured pre-post)
* Transfer and preparation for future learning (measured at post-test)

=== Planned Experiments ===

A comparison of the three conditions will be conducted in an in-vivo school study.
We will control for time, and assign students randomly to conditions.

=== Explanation ===

=== Further Information ===

=== Connections ===
[[Baker_Choices_in_LE_Space]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58

Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.

Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A.,
Mahadevan, S., and Woolf. B.P. (2007) Repairing Disengagement with Non-
Invasive Interventions. Proceedings of the 13h International Conference on
Artificial Intelligence in Education, 195-202.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S.E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.

Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.

Kaminski, J.A., Sloutsky, V.M., Heckler, A. (2009) Transfer of Mathematical Knowledge: The Portability of Generic Instantiations. Child Development, 3 (3), 151-155.

Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While
Providing Proactive Help. Proc. of the International Conference on Artificial Intelligence in
Education, 887-889.

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]

Roll, I., Aleven, V., McLaren, B.M., and Koedinger, K.R. (2007) Can help seeking be
tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of
the 13th International Conference on Artificial Intelligence in Education, 203-210.

Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.

Walonoski, J.A., Heffernan, N.T. (2006) Prevention of Off-Task Gaming Behavior in
Intelligent Tutoring Systems. Proceedings of the 8th International Conference on
Intelligent Tutoring Systems, 722-724.

=== Future Plans ===

Baker - Closing the Loop

2009-12-05T21:38:03Z

Ryan: added lesson name

== Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop” ==
=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker
|-
| '''Other Contributers''' ||
|-
| '''Study Start Date''' || Spring, 2010
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===

This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC ([[Baker_Choices_in_LE_Space]]) (Baker et al, 2009), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1 -- full details on this model are given on the page [[Baker_Choices_in_LE_Space]]) was created, and used to study why some tutor lessons are gamed more than others in the Algebra tutor. The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed (CTA 12: Systems of Equations A), and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

=== Background & Significance ===

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as Help Abuse (Aleven & Koedinger, 2001), and Systematic Guessing (Baker et al, 2004). This pair of strategies is referred to as [[Gaming the system]] (Baker et al, 2004). Gaming the system is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004).

Recent work has indicated that a variety of aspects of cognitive tutor lessons are predictive of greater quantities of gaming (Baker et al, 2009). In particular, gaming is predicted by (boldface indicates a particularly strong relationship -- r2>0.15)

* '''Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest'''
* '''Not immediately apparent what icons in toolbar mean'''
* '''The lesson is not an equation-solver unit'''
* '''The location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated'''
* The same number being used for multiple constructs
* Reading hints does not positively influence performance on future opportunities to use skill
* Proportion of hints in each hint sequence that refer to abstract principles
* Hints do not give directional feedback such as “try a larger number”
* Hint requests that student perform some action

Historically, work to address gaming the system has either attempted to eliminate gaming behavior by making it more difficult to game (for instance, by putting delays between hints), or has attempted to detect gaming automatically and respond to it. The first approach appears to lead students to find new gaming strategies (Murray & VanLehn, 2005), and the second approach has led to systems that reduce gaming (Arroyo et al, 2007; Baker et al, 2006; Roll et al, 2007; Walonoski & Heffernan, 2006) and improve learning (Arroyo et al, 2007; Baker et al, 2006), but has typically been time-consuming and difficult to scale.

Within this project, we propose to use the findings from Baker et al (2009), which found tutor lesson features associated with gaming, as design principles for how to develop tutor lessons that students do not choose to game, in a completely unnoticeable way.

This study will also give us the opportunity to make a contribution to another area of research – recent work has suggested that interest-increasing text may reduce the transferability of knowledge (Kaminski et al, 2009). This may form a trade-off, where interest-increasing text reduces gaming (improving learning) but reduces the transferability of the learning. We will investigate this question with conditions that both include interest-increasing text, and lack interest-increasing text, but are identical in all other ways.

A third potential benefit of this study is in elucidating the link between these aspects of tutors and gaming behavior. We hypothesize that the mediating link is via the affective states of boredom and confusion. We will validate this hypothesis by observing student affect in each condition of the study.

=== Glossary ===

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

=== Hypotheses ===

;H1
: Re-designing a tutor lesson to eliminate features shown to be associated with gaming (see above), will result in lower gaming and better learning

;H2
: The tutor lesson features associated with gaming(see above) influence gaming by increasing the incidence of boredom and confusion

;H3
: Adding interest-increasing text to a tutor lesson (one such feature) will reduce gaming and improve learning, but potentially at the cost of lower transfer

=== Independent Variables ===

Three conditions will be compared:

* An unmodified version of a highly-gamed lesson in Cognitive Tutor Algebra (CTA 12: Systems of Equations A)
* A modified version of the same lesson
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature
* A second modified version of the same lesson
** Adding interest-increasing extraneous text to the scenario
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature

=== Dependent Variables ===

* Incidence of gaming behavior (measured via quantitative field observations -- cf. Baker et al, 2004)
* Incidence of boredom and confusion (measured via quantitative field observations -- cf. Rodrigo et al, 2007)
* Learning of domain skills and concepts (measured pre-post)
* Transfer and preparation for future learning (measured at post-test)

=== Planned Experiments ===

A comparison of the three conditions will be conducted in an in-vivo school study.

=== Explanation ===

=== Further Information ===

=== Connections ===
[[Baker_Choices_in_LE_Space]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58

Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.

Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A.,
Mahadevan, S., and Woolf. B.P. (2007) Repairing Disengagement with Non-
Invasive Interventions. Proceedings of the 13h International Conference on
Artificial Intelligence in Education, 195-202.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S.E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.

Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.

Kaminski, J.A., Sloutsky, V.M., Heckler, A. (2009) Transfer of Mathematical Knowledge: The Portability of Generic Instantiations. Child Development, 3 (3), 151-155.

Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While
Providing Proactive Help. Proc. of the International Conference on Artificial Intelligence in
Education, 887-889.

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]

Roll, I., Aleven, V., McLaren, B.M., and Koedinger, K.R. (2007) Can help seeking be
tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of
the 13th International Conference on Artificial Intelligence in Education, 203-210.

Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.

Walonoski, J.A., Heffernan, N.T. (2006) Prevention of Off-Task Gaming Behavior in
Intelligent Tutoring Systems. Proceedings of the 8th International Conference on
Intelligent Tutoring Systems, 722-724.

=== Future Plans ===

Baker - Closing the Loop

2009-12-05T21:36:40Z

Ryan: add ref

== Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop” ==
=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker
|-
| '''Other Contributers''' ||
|-
| '''Study Start Date''' || Spring, 2010
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===

This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC ([[Baker_Choices_in_LE_Space]]) (Baker et al, 2009), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1 -- full details on this model are given on the page [[Baker_Choices_in_LE_Space]]) was created, and used to study why some tutor lessons are gamed more than others in the Algebra tutor. The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed, and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

=== Background & Significance ===

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as Help Abuse (Aleven & Koedinger, 2001), and Systematic Guessing (Baker et al, 2004). This pair of strategies is referred to as [[Gaming the system]] (Baker et al, 2004). Gaming the system is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004).

Recent work has indicated that a variety of aspects of cognitive tutor lessons are predictive of greater quantities of gaming (Baker et al, 2009). In particular, gaming is predicted by (boldface indicates a particularly strong relationship -- r2>0.15)

* '''Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest'''
* '''Not immediately apparent what icons in toolbar mean'''
* '''The lesson is not an equation-solver unit'''
* '''The location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated'''
* The same number being used for multiple constructs
* Reading hints does not positively influence performance on future opportunities to use skill
* Proportion of hints in each hint sequence that refer to abstract principles
* Hints do not give directional feedback such as “try a larger number”
* Hint requests that student perform some action

Historically, work to address gaming the system has either attempted to eliminate gaming behavior by making it more difficult to game (for instance, by putting delays between hints), or has attempted to detect gaming automatically and respond to it. The first approach appears to lead students to find new gaming strategies (Murray & VanLehn, 2005), and the second approach has led to systems that reduce gaming (Arroyo et al, 2007; Baker et al, 2006; Roll et al, 2007; Walonoski & Heffernan, 2006) and improve learning (Arroyo et al, 2007; Baker et al, 2006), but has typically been time-consuming and difficult to scale.

Within this project, we propose to use the findings from Baker et al (2009), which found tutor lesson features associated with gaming, as design principles for how to develop tutor lessons that students do not choose to game, in a completely unnoticeable way.

This study will also give us the opportunity to make a contribution to another area of research – recent work has suggested that interest-increasing text may reduce the transferability of knowledge (Kaminski et al, 2009). This may form a trade-off, where interest-increasing text reduces gaming (improving learning) but reduces the transferability of the learning. We will investigate this question with conditions that both include interest-increasing text, and lack interest-increasing text, but are identical in all other ways.

A third potential benefit of this study is in elucidating the link between these aspects of tutors and gaming behavior. We hypothesize that the mediating link is via the affective states of boredom and confusion. We will validate this hypothesis by observing student affect in each condition of the study.

=== Glossary ===

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

=== Hypotheses ===

;H1
: Re-designing a tutor lesson to eliminate features shown to be associated with gaming (see above), will result in lower gaming and better learning

;H2
: The tutor lesson features associated with gaming(see above) influence gaming by increasing the incidence of boredom and confusion

;H3
: Adding interest-increasing text to a tutor lesson (one such feature) will reduce gaming and improve learning, but potentially at the cost of lower transfer

=== Independent Variables ===

Three conditions will be compared:

* An unmodified version of a highly-gamed lesson in Cognitive Tutor Algebra
* A modified version of the same lesson
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature
* A second modified version of the same lesson
** Adding interest-increasing extraneous text to the scenario
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature

=== Dependent Variables ===

* Incidence of gaming behavior (measured via quantitative field observations -- cf. Baker et al, 2004)
* Incidence of boredom and confusion (measured via quantitative field observations -- cf. Rodrigo et al, 2007)
* Learning of domain skills and concepts (measured pre-post)
* Transfer and preparation for future learning (measured at post-test)

=== Planned Experiments ===

A comparison of the three conditions will be conducted in an in-vivo school study.

=== Explanation ===

=== Further Information ===

=== Connections ===
[[Baker_Choices_in_LE_Space]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58

Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.

Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A.,
Mahadevan, S., and Woolf. B.P. (2007) Repairing Disengagement with Non-
Invasive Interventions. Proceedings of the 13h International Conference on
Artificial Intelligence in Education, 195-202.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S.E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.

Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.

Kaminski, J.A., Sloutsky, V.M., Heckler, A. (2009) Transfer of Mathematical Knowledge: The Portability of Generic Instantiations. Child Development, 3 (3), 151-155.

Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While
Providing Proactive Help. Proc. of the International Conference on Artificial Intelligence in
Education, 887-889.

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]

Roll, I., Aleven, V., McLaren, B.M., and Koedinger, K.R. (2007) Can help seeking be
tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of
the 13th International Conference on Artificial Intelligence in Education, 203-210.

Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.

Walonoski, J.A., Heffernan, N.T. (2006) Prevention of Off-Task Gaming Behavior in
Intelligent Tutoring Systems. Proceedings of the 8th International Conference on
Intelligent Tutoring Systems, 722-724.

=== Future Plans ===

Baker - Closing the Loop

2009-12-05T21:34:06Z

Ryan: added refs

== Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop” ==
=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker
|-
| '''Other Contributers''' ||
|-
| '''Study Start Date''' || Spring, 2010
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===

This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC ([[Baker_Choices_in_LE_Space]]) (Baker et al, 2009), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1 -- full details on this model are given on the page [[Baker_Choices_in_LE_Space]]) was created, and used to study why some tutor lessons are gamed more than others in the Algebra tutor. The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed, and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

=== Background & Significance ===

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as Help Abuse (Aleven & Koedinger, 2001), and Systematic Guessing (Baker et al, 2004). This pair of strategies is referred to as [[Gaming the system]] (Baker et al, 2004). Gaming the system is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004).

Recent work has indicated that a variety of aspects of cognitive tutor lessons are predictive of greater quantities of gaming (Baker et al, 2009). In particular, gaming is predicted by (boldface indicates a particularly strong relationship -- r2>0.15)

* '''Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest'''
* '''Not immediately apparent what icons in toolbar mean'''
* '''The lesson is not an equation-solver unit'''
* '''The location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated'''
* The same number being used for multiple constructs
* Reading hints does not positively influence performance on future opportunities to use skill
* Proportion of hints in each hint sequence that refer to abstract principles
* Hints do not give directional feedback such as “try a larger number”
* Hint requests that student perform some action

Historically, work to address gaming the system has either attempted to eliminate gaming behavior by making it more difficult to game (for instance, by putting delays between hints), or has attempted to detect gaming automatically and respond to it. The first approach appears to lead students to find new gaming strategies (Murray & VanLehn, 2005), and the second approach has led to systems that reduce gaming (Arroyo et al, 2007; Baker et al, 2006; Roll et al, 2007; Walonoski & Heffernan, 2006) and improve learning (Arroyo et al, 2007; Baker et al, 2006), but has typically been time-consuming and difficult to scale.

Within this project, we propose to use the findings from Baker et al (2009), which found tutor lesson features associated with gaming, as design principles for how to develop tutor lessons that students do not choose to game, in a completely unnoticeable way.

This study will also give us the opportunity to make a contribution to another area of research – recent work has suggested that interest-increasing text may reduce the transferability of knowledge. This may form a trade-off, where interest-increasing text both reduces gaming (improving learning) but reduces the transferability of the learning. We will investigate this question with conditions that both include interest-increasing text, and lack interest-increasing text, but are identical in all other ways.

A third potential benefit of this study is in elucidating the link between these aspects of tutors and gaming behavior. We hypothesize that the mediating link is via the affective states of boredom and confusion. We will validate this hypothesis by observing student affect in each condition of the study.

=== Glossary ===

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

=== Hypotheses ===

;H1
: Re-designing a tutor lesson to eliminate features shown to be associated with gaming (see above), will result in lower gaming and better learning

;H2
: The tutor lesson features associated with gaming(see above) influence gaming by increasing the incidence of boredom and confusion

;H3
: Adding interest-increasing text to a tutor lesson (one such feature) will reduce gaming and improve learning, but potentially at the cost of lower transfer

=== Independent Variables ===

Three conditions will be compared:

* An unmodified version of a highly-gamed lesson in Cognitive Tutor Algebra
* A modified version of the same lesson
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature
* A second modified version of the same lesson
** Adding interest-increasing extraneous text to the scenario
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature

=== Dependent Variables ===

* Incidence of gaming behavior (measured via quantitative field observations -- cf. Baker et al, 2004)
* Incidence of boredom and confusion (measured via quantitative field observations -- cf. Rodrigo et al, 2007)
* Learning of domain skills and concepts (measured pre-post)
* Transfer and preparation for future learning (measured at post-test)

=== Planned Experiments ===

A comparison of the three conditions will be conducted in an in-vivo school study.

=== Explanation ===

=== Further Information ===

=== Connections ===
[[Baker_Choices_in_LE_Space]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58

Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.

Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A.,
Mahadevan, S., and Woolf. B.P. (2007) Repairing Disengagement with Non-
Invasive Interventions. Proceedings of the 13h International Conference on
Artificial Intelligence in Education, 195-202.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S.E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.

Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.

Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While
Providing Proactive Help. Proc. of the International Conference on Artificial Intelligence in
Education, 887-889.

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]

Roll, I., Aleven, V., McLaren, B.M., and Koedinger, K.R. (2007) Can help seeking be
tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of
the 13th International Conference on Artificial Intelligence in Education, 203-210.

Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.

Walonoski, J.A., Heffernan, N.T. (2006) Prevention of Off-Task Gaming Behavior in
Intelligent Tutoring Systems. Proceedings of the 8th International Conference on
Intelligent Tutoring Systems, 722-724.

=== Future Plans ===

Baker - Closing the Loop

2009-12-05T21:30:33Z

Ryan: added refs

== Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop” ==
=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker
|-
| '''Other Contributers''' ||
|-
| '''Study Start Date''' || Spring, 2010
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===

This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC ([[Baker_Choices_in_LE_Space]]) (Baker et al, 2009), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1 -- full details on this model are given on the page [[Baker_Choices_in_LE_Space]]) was created, and used to study why some tutor lessons are gamed more than others in the Algebra tutor. The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed, and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

=== Background & Significance ===

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as Help Abuse (Aleven & Koedinger, 2001), and Systematic Guessing (Baker et al, 2004). This pair of strategies is referred to as [[Gaming the system]] (Baker et al, 2004). Gaming the system is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004).

Recent work has indicated that a variety of aspects of cognitive tutor lessons are predictive of greater quantities of gaming (Baker et al, 2009). In particular, gaming is predicted by (boldface indicates a particularly strong relationship -- r2>0.15)

* '''Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest'''
* '''Not immediately apparent what icons in toolbar mean'''
* '''The lesson is not an equation-solver unit'''
* '''The location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated'''
* The same number being used for multiple constructs
* Reading hints does not positively influence performance on future opportunities to use skill
* Proportion of hints in each hint sequence that refer to abstract principles
* Hints do not give directional feedback such as “try a larger number”
* Hint requests that student perform some action

Historically, work to address gaming the system has either attempted to eliminate gaming behavior by making it more difficult to game (for instance, by putting delays between hints), or has attempted to detect gaming automatically and respond to it. The first approach appears to lead students to find new gaming strategies (Murray & VanLehn, 2005), and the second approach has led to systems that reduce gaming (Arroyo et al, 2007; Baker et al, 2006; Roll et al, 2007; Walonoski & Heffernan, 2006) and improve learning (Arroyo et al, 2007; Baker et al, 2006), but has typically been time-consuming and difficult to scale.

Within this project, we propose to use the findings from Baker et al (2009), which found tutor lesson features associated with gaming, as design principles for how to develop tutor lessons that students do not choose to game, in a completely unnoticeable way.

This study will also give us the opportunity to make a contribution to another area of research – recent work has suggested that interest-increasing text may reduce the transferability of knowledge. This may form a trade-off, where interest-increasing text both reduces gaming (improving learning) but reduces the transferability of the learning. We will investigate this question with conditions that both include interest-increasing text, and lack interest-increasing text, but are identical in all other ways.

A third potential benefit of this study is in elucidating the link between these aspects of tutors and gaming behavior. We hypothesize that the mediating link is via the affective states of boredom and confusion. We will validate this hypothesis by observing student affect in each condition of the study.

=== Glossary ===

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

=== Hypotheses ===

;H1
: Re-designing a tutor lesson to eliminate features shown to be associated with gaming (see above), will result in lower gaming and better learning

;H2
: The tutor lesson features associated with gaming(see above) influence gaming by increasing the incidence of boredom and confusion

;H3
: Adding interest-increasing text to a tutor lesson (one such feature) will reduce gaming and improve learning, but potentially at the cost of lower transfer

=== Independent Variables ===

Three conditions will be compared:

* An unmodified version of a highly-gamed lesson in Cognitive Tutor Algebra
* A modified version of the same lesson
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature
* A second modified version of the same lesson
** Adding interest-increasing extraneous text to the scenario
** Clearer communication of the flow of problem-solving through the interface (with a giant arrow)
** Fewer help messages that are wholly abstract in nature

=== Dependent Variables ===

* Incidence of gaming behavior (measured via quantitative field observations -- cf. Baker et al, 2004)
* Incidence of boredom and confusion (measured via quantitative field observations -- cf. Rodrigo et al, 2007)
* Learning of domain skills and concepts (measured pre-post)
* Transfer and preparation for future learning (measured at post-test)

=== Planned Experiments ===

A comparison of the three conditions will be conducted in an in-vivo school study.

=== Explanation ===

=== Further Information ===

=== Connections ===
[[Baker_Choices_in_LE_Space]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58

Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.

Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.

Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While
Providing Proactive Help. Proc. of the International Conference on Artificial Intelligence in
Education, 887-889.

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]

Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.

=== Future Plans ===

Baker Choices in LE Space

2009-12-05T21:26:18Z

Ryan:

== How Content and Interface Features Influence Student Choices Within the Learning Spaces==

Ryan S.J.d. Baker, Albert T. Corbett, Kenneth R. Koedinger, Ma. Mercedes T. Rodrigo

===Overview===

PIs: Ryan S.J.d Baker

Co-PIs: Albert T. Corbett, Kenneth R. Koedinger

Others who have contributed 160 hours or more:

* Jay Raspat, Carnegie Mellon University, taxonomy development
* Adriana M.J.A. de Carvalho, Carnegie Mellon University, data coding

Others significant personnel :

* Ma. Mercedes T. Rodrigo, Ateneo de Manila University, data coding methods
* Vincent Aleven, Carnegie Mellon University, taxonomy development

===Abstract===

We are investigating what factors lead students to make specific path choices in the learning space, focusing specifically on the shallow strategy known as [[gaming the system]], and on [[Off-Task Behavior]]. Prior PSLC research has shown that a variety of motivations, attitudes, and affective states are associated with the choice to game the system (Baker et al, 2004; Baker, 2007b; Rodrigo et al, 2007) and the choice of off-task behavior (Baker, 2007b) within intelligent tutoring systems. However, other recent research has found that differences between lessons are on the whole better predictors of gaming than differences between students (Baker, 2007), suggesting that contextual factors associated with a specific tutor unit may be the most important reason why students game the system. Hence, this project is investigating how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy. An extension to this project in 2008-2009 also investigated how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy.

To this end, we have annotated a large proportion of the learning events/transactions in a set of twenty units in the [[Algebra]] LearnLab with descriptions of each unit's content and interface features, using a combination of human coding and educational data mining. We then used data mining to predict gaming and off-task behavior with the content and interface features of the units they occur in. This gives us new insight into why students make specific path choices in the learning space, and explains the prior finding that path choices differ considerably between tutor units.

===Glossary===

*[[Gaming the system]]
*[[Help abuse]]
*[[Systematic Guessing]]
*[[Off-Task Behavior]]

===Research Questions===

What aspects of tutor lesson design lead to the choice to game the system?

What aspects of tutor lesson design lead to the choice to go off-task?

===Hypothesis===

;H1
:Content or interface features better explain differences in gaming frequency than stable between-student differences

;H2
:Specific content or interface features will be replicably associated with differences in gaming the system across students

;H3
:Specific content or interface features will be replicably associated with differences in off-task behavior across students

===Background and Significance===

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as [[Help Abuse]] (Aleven & Koedinger, 2001), [[Systematic Guessing]] (Baker et al, 2004), and the failure to engage in [[Self-explanation]]. A student may also leave the learning event space entirely by engaging in various forms of off-task behavior.

One analytical tool with considerable power to help learning scientists explain the ways students choose to use a learning environment is the [[learning event space]]. In a learning event space, the different paths a student could take are enumerated, and the effects of each path are detailed, both in terms of how the path influences the student’s success within the environment, and the student’s learning. The learning event space model provides a simple way to identify the possible paths and effects; it also provides a concrete way to break down complex research questions into simpler and more concrete questions.

[[Gaming the system]] is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004). It was earlier hypothesized that gaming stemmed from stable differences in student goals, motivation, and attitudes -- however multiple studies have now suggested that these constructs play only a small role in predicting gaming behavior (Baker et al, 2005; Walonoski & Heffernan, 2006; Baker et al, 2008). By contrast, variation in short-term affective states and the tutor lesson itself appear to play a much larger role in the choice to game (Rodrigo et al, 2007; Baker, 2007a).

In this project, we investigate what it is about some tutor lessons that encourages or discourages gaming. This project helps explain why students choose shallow gaming strategies at some learning events and not at others. This contributes to our understanding of learning event spaces, and makes a significant contribution to the PSLC Theoretical Framework, by providing an account for why students choose the shallow learning strategies in many of the learning event space models in the PSLC Theoretical Framework. The study of what lesson features predicted gaming was anticipated to jump-start the process of studying why students choose other shallow learning strategies beyond gaming the system, by providing a methodological template that can be directly applied in future research, as well as initial hypotheses to investigate. It did so, enabling analysis of which lesson features are associated with the choice to go off-task. This study has influenced the upcoming PSLC project [[Baker Closing the Loop on Gaming]].

===Independent Variables===

We have developed a taxonomy for how Cognitive Tutor lessons can differ from one another, the Cognitive Tutor Lesson Variation Space, version 1.1 (CTLVS1.1). The CTLVS1 was developed by a six member design team with a variety of perspectives and expertise, including three Cognitive Tutor designers (with expertise in cognitive psychology and artificial intelligence), a researcher specializing in the study of gaming the system, a mathematics teacher with several years of experience using Cognitive Tutors in class, and a designer of non-computerized curricula who had not previously used a Cognitive Tutor. Full detail on the CTLVS1's design is given in Baker et al (in press a).

The CTLVS1's features are as follows:

'''Difficulty, Complexity of Material, and Time-Consumingness'''
* 1. Average percent error
* 2. Lesson consists solely of review of material encountered in previous lessons
* 3. Average probability that student will learn a skill at each opportunity to practice skill (cf. Corbett & Anderson, 1995)
* 4. Average initial probability that student will know a skill when starting tutor (cf. Corbett & Anderson, 1995)
* 5. Average number of extraneous “distractor” values per problem
* 6. Proportion of problems where extraneous “distractor” values are given
* 7. Maximum number of mathematical operators needed to give correct answer on any step in lesson
* 8. Maximum number of mathematical operators mentioned in hint on any step in lesson
* 9. Intermediate calculations must be done outside of software (mentally or on paper) for some problem steps (ever occurs)
* 10. Proportion of hints that discuss intermediate calculations that must be done outside of software (mentally or on paper)
* 11. Total number of skills in lesson
* 12. Average time per problem step
* 13. Proportion of problem statements that incorporate multiple representations (for example: diagram as well as text)
* 14. Proportion of problem statements that use same numeric value for two constructs
* 15. Average number of distinct/separable questions or problem-solving tasks per problem
* 16. Maximum number of distinct/separable questions or problem-solving tasks in any problem
* 17. Average number of numerical quantities manipulated per step
* 18. Average number of times each skill is repeated per problem
* 19. Number of problems in lesson
* 20. Average time spent in lesson
* 21. Average number of problem steps per problem
* 22. Minimum number of answers or interface actions required to complete problem

'''Quality of Help Features'''
* 23. Average amount that reading on-demand hints improves performance on future opportunities to use skill (cf. Beck, 2006)
* 24. Average Flesch-Kincaid Grade Reading Level of hints
* 25. Proportion of hints using inductive support, going from example to abstract description of concept/principle (Koedinger & Anderson, 1998)
* 26. Proportion of hints that explicitly explain concepts or principles underlying current problem-solving step
* 27. Proportion of hints that explicitly refer to abstract principles
* 28. On average, how many hints must student request before concrete features of problems are discussed
* 29. Average number of hint messages per hint sequence that orient student to mathematical sub-goal
* 30. Proportion of hints that explicitly refer to scenario content (instead of referring solely to mathematical constructs)
* 31. Proportion of hint sequences that use terminology specific to this software
* 32. Proportion of hint messages which refer solely to interface features
* 33. Proportion of hint messages that cannot be understood by teacher
* 34. Proportion of hint messages with complex noun phrases
* 35. Proportion of skills where the only hint message explicitly tells student what to do

'''Usability'''
* 36. First problem step in first problem of lesson is either clearly indicated, or follows established convention (such as top-left cell in worksheet)
* 37. Problem-solving task in lesson is not made immediately clear
* 38. After student completes step, system indicates where in interface next action should occur
* 39. Proportion of steps where it is necessary to request hint to figure out what to do next
* 40. Not immediately apparent what icons in toolbar mean
* 41. Screen is sufficiently cluttered with interface widgets, that it is difficult to determine where to enter answers
* 42. Proportion of steps where student must change a value in a cell that was previously treated as correct (examples: self-detection of errors; refinement of answers)
* 43. Format of answer changes between problem steps without clear indication
* 44. If student has skipped step, and asks for hint, hints refer to skipped step without explicitly highlighting in interface (ever seen)
* 45. If student has skipped step, and asks for hint, skipped step is explicitly highlighted in interface (ever seen)

'''Relevance and Interestingness'''
* 46. Proportion of problem statements which involve concrete people/places/things, rather than just numerical quantities
* 47. Proportion of problem statements with story content
* 48. Proportion of problem statements which involve scenarios relevant to the "World of Work"
* 49. Proportion of problem statements which involve scenarios relevant to students’ current daily life
* 50. Proportion of problem statements which involve fantasy (example: being a rock star)
* 51. Proportion of problem statements which involve concrete details unfamiliar to population of students (example: dog-sleds)
* 52. Proportion of problems which use (or appear to use) genuine data
* 53. Proportion of problem statements with text not directly related to problem-solving task
* 54. Average number of person proper names in problem statements

'''Aspects of “buggy” messages notifying student why action was incorrect'''
* 55. Proportion of buggy messages that indicate which concept student demonstrated misconception in
* 56. Proportion of buggy messages that indicate how student’s action was the result of a procedural error
* 57. Proportion of buggy messages that refer solely to interface action
* 58. Buggy messages are not immediately given; instead icon appears, which can be hovered over to receive bug message

'''Design Choices Which Make It Easier to Game the System'''
* 59. Proportion of steps which are explicitly multiple-choice
* 60. Average number of choices in multiple-choice step
* 61. Proportion of hint sequences with final hint that explicitly tells student what the answer is, but not what/how to enter it in the tutor software
* 62. Hint gives directional feedback (example: “try a larger number”) (ever seen)
* 63. Average number of feasible answers for each problem step

'''Meta-Cognition and Complex Conceptual Thinking (or features that make them easy to avoid)'''
* 64. Student is prompted to give [[self-explanations]]
* 65. Hints give explicit metacognitive advice (ever seen)
* 66. Proportion of problem statements that use common word to indicate mathematical operation to use (example: “increase”)
* 67. Proportion of problem statements that indicate mathematical operation to use, but with uncommon terminology (example: “pounds below normal” to indicate subtraction)
* 68. Proportion of problem statements that explicitly tell student which mathematical operation to use (example: “add”)

'''Software Bugs/Implementation Flaws (rare)'''
* 69. Percent of problems where grammatical error is found in problem statement
* 70. Reference in problem statement to interface component that does not exist (ever occurs)
* 71. Proportion of problem steps where hints are unavailable
* 72. Hint recommends student do something which is incorrect or non-optimal (ever occurs)
* 73. Student can advance to new problem despite still visible errors on intermediate problem-solving steps

'''Miscellaneous'''
* 74. Hint requests that student perform some action
* 75. Value of answer is very large (over four significant digits) (ever seen)
* 76. Average length of text in multiple-choice popup widgets
* 77. Proportion of problem statements which include question or imperative
* 78. Student selects action from menu, tutor software performs action (as opposed to typing in answers, or direct manipulation)
* 79. Lesson is an "equation-solver" unit

We then labeled a large proportion of units
in the [[Algebra]] LearnLab with these taxonomic features. These features
make up the independent variables in this project.

===Dependent Variables===

We labeled approximately 1.2 million transactions in [[Algebra]] tutor data from the [[DataShop]] with predictions as to whether it is an instance of gaming the system.
These predictions were created by using text replay observations (Baker, Corbett, & Wagner, 2006) to label a representative set of transactions, and then using these labels to create gaming detectors (cf. Baker, Corbett, & Koedinger, 2004; Baker et al, 2008) which can be used to label the remaining transactions.

===Findings and Explanation===

The text below is taken from (Baker, 2007b; Baker et al, in press a, accepted).

The difference between lessons is a significantly better predictor than the difference between students in determining how much gaming behavior a student will engage in, in a given lesson. Put more simply, knowing which lesson a student is using is a better predictor of how much gaming will occur, than knowing which student it is.

In the Middle School Tutor, lesson has 35 parameters and achieves an r-squared of 0.55. Student has 240 parameters and achieves an r-squared of 0.16. In the Algebra Tutor, lesson has 21 parameters and achieves an r-squared of 0.18. Student achieves an equal r-squared, but with 58 students; hence, lesson is a statistically better predictor because it achieves equal or significantly better fit with considerably fewer parameters.

We empirically grouped the 79 features of the CTLVS1.1 with Principal Component Analysis (PCA). We grouped the 79 features of the CTLVS1 into 6 factors. We then analyzed whether the correlation between these 6 factorsand the frequency of gaming the system was significant in any case.

Of these 6 factors, one was statistically significantly associated with the choice to game the system, r2 = 0.29 (e.g. accounting for 29% of the variance in gaming), F(1,19)= 7.84, p=0.01. The factor loaded strongly on eight features associated with more gaming:
* 14: The same number being used for multiple constructs
* 23-inverse-direction: Reading hints does not positively influence performance on future opportunities to use skill
* 27: Proportion of hints in each hint sequence that refer to abstract principles
* 40: Not immediately apparent what icons in toolbar mean
* 53-inverse-direction: Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest
* 63-inverse-direction: Hints do not give directional feedback such as “try a larger number”
* 71-inverse-direction: Lack of implementation flaw in hint message, where there is a reference to a non-existent interface component
* 75: Hint requests that student perform some action

In general, several of the features in this factor appear to correspond to a lack of clarity in the presentation of the content or task (23-inverse, 40, 63-inverse), as well as abstractness (27) and ambiguity (14). Curiously, feature 71-inverse (the lack of a specific type of implementation flaw in hint messages, which would make things very unclear) appears to point in the opposite direction – however, this implementation flaw was only common in a single rarely gamed lesson, so this result is probably a statistical artifact.

Feature 53-inverse appears to represent a different construct – interestingness (or the attempt to increase interestingness). The fact that feature 53 was associated with less gaming whereas more specific interest-increasing features (features 46-52) were not so strongly related may suggest that it is less important exactly how a problem scenario attempts to increase interest, than it is important that the problem scenario has some content in it that is not strictly mathematical.

Taken individually, two of the constructs in this factor were significantly (or marginally significantly) associated with gaming. Feature 53-inverse (text in the problem statement not directly related to the problem-solving task) was associated with significantly less gaming, r2 = 0.19, F(1,19) = 4.59, p = 0.04. Feature 40 (when it is not immediately apparent what icons in toolbar mean) was marginally significantly associated with more gaming, r2 = 0.15, F(1, 19)=3.52, p=0.08. The fact that other top features in the factor were not independently associated with gaming, while the factor as a whole was fairly strongly associated with gaming, suggests that gaming may occur primarily when more than one of these features are present.

Two features that were not present in the significant factor was statistically significantly associated with gaming: Feature 36, where the location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated, r2 = 0.20, F(1,19)=4.97, p=0.04. This feature, like many of those in the gaming-related factor, represents an unclear or confusing lesson. Also, Feature 79, whether or not the lesson was an equation solver unit, was statistically significantly better than chance, r2 = 0.30, F(1, 19)=8.55, p<0.01. Note, however, that although a lower amount of interesting text is generally associated with more gaming (Feature 53), equation-solver units (which have no text) have less gaming in general (Feature 79). This result may suggest that interest-increasing text is only beneficial (for reducing gaming) above a certain threshold -- alternatively, other aspects of the equation-solver units may have reduced gaming even though the lack of interesting-increasing text would generally be expected to increase it.

When the gaming-related factor, Feature 36, and Feature 79, were included in a model together, all remain statistically significant, and the combined model explains 56% of the variance in gaming (e.g. r2 = 0.55).

Five other features that were not strongly loaded in the significant factor were marginally associated with gaming. None of these other features is statistically significant in a model that already includes the gaming-related cluster and Feature 36. Due to the non-conclusiveness of the evidence relevant to these features, we will not discuss all of these features in detail, but will briefly mention one that has appeared in prior discussions of gaming. Lessons where a higher proportion of hint sequences told students what to do on the last hint (Feature 61) had marginally significantly more gaming, r2 = 0.14, F(1,19)=3.28, p=0.09. This result is unsurprising, as drilling through hints and typing in a bottom-out hint is one of the easiest and most frequently reported types of [[gaming the system]].

The off-task behavior model achieved similar predictive power, but was a much less complex model. None of the 6 factors were statistically significantly associated with gaming. Only one of the features was individually statistically significantly associated with off-task behavior: Feature 79, whether or not the lesson was an equation solver unit. Equation solver units had significantly less off-task behavior, just as they had significantly less gaming the system, and the effect was large in magnitude, r2 = 0.55, F(1, 21)=27.29, p<0.001, Bonferroni adjusted p<0.001.

To put this relationship into better context, we can look at the proportion of time students
spent off-task in equation-solver lessons as compared to other lessons. On average,
students spent 4.4% of their time off-task within the equation-solver lessons, much lower
than is generally seen in intelligent tutor classrooms or, for that matter, in traditional
classrooms. By contrast, students spent 14.1% of their time off-task within the
other lessons, a proportion of time-on-task which is much more in line with previous
observations. The difference in time spent per type of lesson is, as would be expected,
statistically significant, t(22)=4.48, p<0.001.

=== Connections to Other PSLC Studies===

This study inspired and led to the upcoming Year 6 study, [[Baker - Closing the Loop]].

===Annotated Bibliography===

===References===
Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58

Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.

Baker, R.S.J.d. (2007a) Is Gaming the System State-or-Trait? Educational Data Mining Through the Multi-Contextual Application of a Validated Behavioral Model. Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, 76-80. [http://www.joazeirodebaker.net/ryan/B2007B.pdf pdf]

Baker, R.S.J.d. (2007b) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S.J.d. (2009) Differences Between Intelligent Tutor Lessons, and the Choice to Go Off-Task. Proceedings of the 2nd International Conference on Educational Data Mining, 11-20.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (2009) Educational Software Features that Encourage and Discourage "Gaming the System". Proceedings of the 14th International Conference on Artificial Intelligence in Education, 475-482.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (2008) Developing a Generalizable Detector of When Students Game the System. User Modeling and User-Adapted Interaction, 18, 3, 287-314. [http://www.joazeirodebaker.net/ryan/USER475.pdf pdf]

Baker, R.S., Corbett, A.T., Wagner, A.Z. (2006) Human Classification of Low-Fidelity Replays of Student Actions. Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems, 29-36. [http://www.joazeirodebaker.net/ryan/BCWFinal.pdf pdf]

Baker, R.S., Roll, I., Corbett, A.T., Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 57-64. [http://www.joazeirodebaker.net/ryan/BRCKAIED2005Final.pdf pdf]

Baker, R.S.J.d., Walonoski, J.A., Heffernan, N.T., Roll, I., Corbett, A.T., Koedinger, K.R. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224. [http://www.joazeirodebaker.net/ryan/BWHRKC-JILR-draft.pdf pdf]

Beck, J.E. (2006) Using Learning Decomposition to Analyze Student Fluency Development. Workshop on Educational Data Mining at the 8th International Conference on Intelligent Tutoring Systems, 21-28.

Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.

Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.

Corbett,A.T., & Anderson, J.R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.

Koedinger, K. R., & Anderson, J. R. (1998). Illustrating principled design: The early evolution of a cognitive tutor for algebra symbolization. Interactive Learning Environments, 5, 161-180.

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]

Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.

Walonoski, J.A., Heffernan, N.T. (2006) Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 382-391.

Baker - Closing the Loop

2009-12-05T21:24:21Z

Ryan:

Baker - Closing the Loop

2009-12-05T21:23:34Z

Ryan: /* Background & Significance */

Baker - Closing the Loop

2009-12-05T21:21:31Z

Ryan: /* Planned Experiments */

Baker - Closing the Loop

2009-12-05T21:20:57Z

Ryan:

Baker - Closing the Loop

2009-12-05T21:20:29Z

Ryan: /* Dependent Variables */ conditions of study

Baker - Closing the Loop

2009-12-05T21:17:47Z

Ryan: /* Independent Variables */

Baker - Closing the Loop

2009-12-05T21:15:13Z

Ryan: /* Background & Significance */

Baker - Closing the Loop

2009-12-05T21:14:17Z

Ryan: /* Hypotheses */

Baker - Closing the Loop

2009-12-05T21:14:09Z

Ryan: /* Hypotheses */

Baker - Closing the Loop

2009-12-05T21:11:55Z

Ryan: /* Background & Significance */

Baker - Closing the Loop

2009-12-05T21:07:26Z

Ryan: /* Background & Significance */

Baker - Closing the Loop

2009-12-05T21:06:28Z

Ryan: /* Background & Significance */ added prior predictors

Baker - Closing the Loop

2009-12-05T21:00:54Z

Ryan: /* Background & Significance */

Baker - Closing the Loop

2009-12-05T20:57:56Z

Ryan: /* Abstract */

Baker - Closing the Loop

2009-12-05T20:56:39Z

Ryan: /* Study 1 */

Baker - Closing the Loop

2009-12-05T20:56:29Z

Ryan: /* Study 1 */

== Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop” ==
=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker
|-
| '''Other Contributers''' ||
|-
| '''Study Start Date''' || Spring, 2010
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===

This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC ([[Baker_Choices_in_LE_Space]]), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1) was created, and used to study why some tutor lessons are gamed more than others (in the Algebra tutor). The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed, and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

=== Background & Significance ===

=== Glossary ===

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

=== Hypotheses ===

;H1
:

;H2
:

;H3
:

;H4
:

=== Independent Variables ===
=== Dependent Variables ===

=== Planned Experiments ===

=== Explanation ===
=== Further Information ===
==== Connections ====
==== Annotated Bibliography ====
==== References ====
==== Future Plans ====

Baker Choices in LE Space

2009-12-05T20:55:35Z

Ryan: /* Connections to Other PSLC Studies */

== How Content and Interface Features Influence Student Choices Within the Learning Spaces==

Ryan S.J.d. Baker, Albert T. Corbett, Kenneth R. Koedinger, Ma. Mercedes T. Rodrigo

===Overview===

PIs: Ryan S.J.d Baker

Co-PIs: Albert T. Corbett, Kenneth R. Koedinger

Others who have contributed 160 hours or more:

* Jay Raspat, Carnegie Mellon University, taxonomy development
* Adriana M.J.A. de Carvalho, Carnegie Mellon University, data coding

Others significant personnel :

* Ma. Mercedes T. Rodrigo, Ateneo de Manila University, data coding methods
* Vincent Aleven, Carnegie Mellon University, taxonomy development

===Abstract===

We are investigating what factors lead students to make specific path choices in the learning space, focusing specifically on the shallow strategy known as [[gaming the system]], and on [[Off-Task Behavior]]. Prior PSLC research has shown that a variety of motivations, attitudes, and affective states are associated with the choice to game the system (Baker et al, 2004; Baker, 2007b; Rodrigo et al, 2007) and the choice of off-task behavior (Baker, 2007b) within intelligent tutoring systems. However, other recent research has found that differences between lessons are on the whole better predictors of gaming than differences between students (Baker, 2007), suggesting that contextual factors associated with a specific tutor unit may be the most important reason why students game the system. Hence, this project is investigating how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy. An extension to this project in 2008-2009 also investigated how the content and presentational/interface aspects of a learning environment influence whether students tend to choose a gaming the system strategy.

To this end, we have annotated a large proportion of the learning events/transactions in a set of twenty units in the [[Algebra]] LearnLab with descriptions of each unit's content and interface features, using a combination of human coding and educational data mining. We then used data mining to predict gaming and off-task behavior with the content and interface features of the units they occur in. This gives us new insight into why students make specific path choices in the learning space, and explains the prior finding that path choices differ considerably between tutor units.

===Glossary===

*[[Gaming the system]]
*[[Help abuse]]
*[[Systematic Guessing]]
*[[Off-Task Behavior]]

===Research Questions===

What aspects of tutor lesson design lead to the choice to game the system?

What aspects of tutor lesson design lead to the choice to go off-task?

===Hypothesis===

;H1
:Content or interface features better explain differences in gaming frequency than stable between-student differences

;H2
:Specific content or interface features will be replicably associated with differences in gaming the system across students

;H3
:Specific content or interface features will be replicably associated with differences in off-task behavior across students

===Background and Significance===

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as [[Help Abuse]] (Aleven & Koedinger, 2001), [[Systematic Guessing]] (Baker et al, 2004), and the failure to engage in [[Self-explanation]]. A student may also leave the learning event space entirely by engaging in various forms of off-task behavior.

One analytical tool with considerable power to help learning scientists explain the ways students choose to use a learning environment is the [[learning event space]]. In a learning event space, the different paths a student could take are enumerated, and the effects of each path are detailed, both in terms of how the path influences the student’s success within the environment, and the student’s learning. The learning event space model provides a simple way to identify the possible paths and effects; it also provides a concrete way to break down complex research questions into simpler and more concrete questions.

[[Gaming the system]] is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004). It was earlier hypothesized that gaming stemmed from stable differences in student goals, motivation, and attitudes -- however multiple studies have now suggested that these constructs play only a small role in predicting gaming behavior (Baker et al, 2005; Walonoski & Heffernan, 2006; Baker et al, 2008). By contrast, variation in short-term affective states and the tutor lesson itself appear to play a much larger role in the choice to game (Rodrigo et al, 2007; Baker, 2007a).

In this project, we investigate what it is about some tutor lessons that encourages or discourages gaming. This project helps explain why students choose shallow gaming strategies at some learning events and not at others. This contributes to our understanding of learning event spaces, and makes a significant contribution to the PSLC Theoretical Framework, by providing an account for why students choose the shallow learning strategies in many of the learning event space models in the PSLC Theoretical Framework. The study of what lesson features predicted gaming was anticipated to jump-start the process of studying why students choose other shallow learning strategies beyond gaming the system, by providing a methodological template that can be directly applied in future research, as well as initial hypotheses to investigate. It did so, enabling analysis of which lesson features are associated with the choice to go off-task. This study has influenced the upcoming PSLC project [[Baker Closing the Loop on Gaming]].

===Independent Variables===

We have developed a taxonomy for how Cognitive Tutor lessons can differ from one another, the Cognitive Tutor Lesson Variation Space, version 1.1 (CTLVS1.1). The CTLVS1 was developed by a six member design team with a variety of perspectives and expertise, including three Cognitive Tutor designers (with expertise in cognitive psychology and artificial intelligence), a researcher specializing in the study of gaming the system, a mathematics teacher with several years of experience using Cognitive Tutors in class, and a designer of non-computerized curricula who had not previously used a Cognitive Tutor. Full detail on the CTLVS1's design is given in Baker et al (in press a).

The CTLVS1's features are as follows:

'''Difficulty, Complexity of Material, and Time-Consumingness'''
* 1. Average percent error
* 2. Lesson consists solely of review of material encountered in previous lessons
* 3. Average probability that student will learn a skill at each opportunity to practice skill (cf. Corbett & Anderson, 1995)
* 4. Average initial probability that student will know a skill when starting tutor (cf. Corbett & Anderson, 1995)
* 5. Average number of extraneous “distractor” values per problem
* 6. Proportion of problems where extraneous “distractor” values are given
* 7. Maximum number of mathematical operators needed to give correct answer on any step in lesson
* 8. Maximum number of mathematical operators mentioned in hint on any step in lesson
* 9. Intermediate calculations must be done outside of software (mentally or on paper) for some problem steps (ever occurs)
* 10. Proportion of hints that discuss intermediate calculations that must be done outside of software (mentally or on paper)
* 11. Total number of skills in lesson
* 12. Average time per problem step
* 13. Proportion of problem statements that incorporate multiple representations (for example: diagram as well as text)
* 14. Proportion of problem statements that use same numeric value for two constructs
* 15. Average number of distinct/separable questions or problem-solving tasks per problem
* 16. Maximum number of distinct/separable questions or problem-solving tasks in any problem
* 17. Average number of numerical quantities manipulated per step
* 18. Average number of times each skill is repeated per problem
* 19. Number of problems in lesson
* 20. Average time spent in lesson
* 21. Average number of problem steps per problem
* 22. Minimum number of answers or interface actions required to complete problem

'''Quality of Help Features'''
* 23. Average amount that reading on-demand hints improves performance on future opportunities to use skill (cf. Beck, 2006)
* 24. Average Flesch-Kincaid Grade Reading Level of hints
* 25. Proportion of hints using inductive support, going from example to abstract description of concept/principle (Koedinger & Anderson, 1998)
* 26. Proportion of hints that explicitly explain concepts or principles underlying current problem-solving step
* 27. Proportion of hints that explicitly refer to abstract principles
* 28. On average, how many hints must student request before concrete features of problems are discussed
* 29. Average number of hint messages per hint sequence that orient student to mathematical sub-goal
* 30. Proportion of hints that explicitly refer to scenario content (instead of referring solely to mathematical constructs)
* 31. Proportion of hint sequences that use terminology specific to this software
* 32. Proportion of hint messages which refer solely to interface features
* 33. Proportion of hint messages that cannot be understood by teacher
* 34. Proportion of hint messages with complex noun phrases
* 35. Proportion of skills where the only hint message explicitly tells student what to do

'''Usability'''
* 36. First problem step in first problem of lesson is either clearly indicated, or follows established convention (such as top-left cell in worksheet)
* 37. Problem-solving task in lesson is not made immediately clear
* 38. After student completes step, system indicates where in interface next action should occur
* 39. Proportion of steps where it is necessary to request hint to figure out what to do next
* 40. Not immediately apparent what icons in toolbar mean
* 41. Screen is sufficiently cluttered with interface widgets, that it is difficult to determine where to enter answers
* 42. Proportion of steps where student must change a value in a cell that was previously treated as correct (examples: self-detection of errors; refinement of answers)
* 43. Format of answer changes between problem steps without clear indication
* 44. If student has skipped step, and asks for hint, hints refer to skipped step without explicitly highlighting in interface (ever seen)
* 45. If student has skipped step, and asks for hint, skipped step is explicitly highlighted in interface (ever seen)

'''Relevance and Interestingness'''
* 46. Proportion of problem statements which involve concrete people/places/things, rather than just numerical quantities
* 47. Proportion of problem statements with story content
* 48. Proportion of problem statements which involve scenarios relevant to the "World of Work"
* 49. Proportion of problem statements which involve scenarios relevant to students’ current daily life
* 50. Proportion of problem statements which involve fantasy (example: being a rock star)
* 51. Proportion of problem statements which involve concrete details unfamiliar to population of students (example: dog-sleds)
* 52. Proportion of problems which use (or appear to use) genuine data
* 53. Proportion of problem statements with text not directly related to problem-solving task
* 54. Average number of person proper names in problem statements

'''Aspects of “buggy” messages notifying student why action was incorrect'''
* 55. Proportion of buggy messages that indicate which concept student demonstrated misconception in
* 56. Proportion of buggy messages that indicate how student’s action was the result of a procedural error
* 57. Proportion of buggy messages that refer solely to interface action
* 58. Buggy messages are not immediately given; instead icon appears, which can be hovered over to receive bug message

'''Design Choices Which Make It Easier to Game the System'''
* 59. Proportion of steps which are explicitly multiple-choice
* 60. Average number of choices in multiple-choice step
* 61. Proportion of hint sequences with final hint that explicitly tells student what the answer is, but not what/how to enter it in the tutor software
* 62. Hint gives directional feedback (example: “try a larger number”) (ever seen)
* 63. Average number of feasible answers for each problem step

'''Meta-Cognition and Complex Conceptual Thinking (or features that make them easy to avoid)'''
* 64. Student is prompted to give [[self-explanations]]
* 65. Hints give explicit metacognitive advice (ever seen)
* 66. Proportion of problem statements that use common word to indicate mathematical operation to use (example: “increase”)
* 67. Proportion of problem statements that indicate mathematical operation to use, but with uncommon terminology (example: “pounds below normal” to indicate subtraction)
* 68. Proportion of problem statements that explicitly tell student which mathematical operation to use (example: “add”)

'''Software Bugs/Implementation Flaws (rare)'''
* 69. Percent of problems where grammatical error is found in problem statement
* 70. Reference in problem statement to interface component that does not exist (ever occurs)
* 71. Proportion of problem steps where hints are unavailable
* 72. Hint recommends student do something which is incorrect or non-optimal (ever occurs)
* 73. Student can advance to new problem despite still visible errors on intermediate problem-solving steps

'''Miscellaneous'''
* 74. Hint requests that student perform some action
* 75. Value of answer is very large (over four significant digits) (ever seen)
* 76. Average length of text in multiple-choice popup widgets
* 77. Proportion of problem statements which include question or imperative
* 78. Student selects action from menu, tutor software performs action (as opposed to typing in answers, or direct manipulation)
* 79. Lesson is an "equation-solver" unit

We then labeled a large proportion of units
in the [[Algebra]] LearnLab with these taxonomic features. These features
make up the independent variables in this project.

===Dependent Variables===

We labeled approximately 1.2 million transactions in [[Algebra]] tutor data from the [[DataShop]] with predictions as to whether it is an instance of gaming the system.
These predictions were created by using text replay observations (Baker, Corbett, & Wagner, 2006) to label a representative set of transactions, and then using these labels to create gaming detectors (cf. Baker, Corbett, & Koedinger, 2004; Baker et al, 2008) which can be used to label the remaining transactions.

===Findings and Explanation===

The text below is taken from (Baker, 2007b; Baker et al, in press a, accepted).

The difference between lessons is a significantly better predictor than the difference between students in determining how much gaming behavior a student will engage in, in a given lesson. Put more simply, knowing which lesson a student is using is a better predictor of how much gaming will occur, than knowing which student it is.

In the Middle School Tutor, lesson has 35 parameters and achieves an r-squared of 0.55. Student has 240 parameters and achieves an r-squared of 0.16. In the Algebra Tutor, lesson has 21 parameters and achieves an r-squared of 0.18. Student achieves an equal r-squared, but with 58 students; hence, lesson is a statistically better predictor because it achieves equal or significantly better fit with considerably fewer parameters.

We empirically grouped the 79 features of the CTLVS1.1 with Principal Component Analysis (PCA). We grouped the 79 features of the CTLVS1 into 6 factors. We then analyzed whether the correlation between these 6 factorsand the frequency of gaming the system was significant in any case.

Of these 6 factors, one was statistically significantly associated with the choice to game the system, r2 = 0.29 (e.g. accounting for 29% of the variance in gaming), F(1,19)= 7.84, p=0.01. The factor loaded strongly on eight features associated with more gaming:
* 14: The same number being used for multiple constructs
* 23-inverse-direction: Reading hints does not positively influence performance on future opportunities to use skill
* 27: Proportion of hints in each hint sequence that refer to abstract principles
* 40: Not immediately apparent what icons in toolbar mean
* 53-inverse-direction: Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest
* 63-inverse-direction: Hints do not give directional feedback such as “try a larger number”
* 71-inverse-direction: Lack of implementation flaw in hint message, where there is a reference to a non-existent interface component
* 75: Hint requests that student perform some action

In general, several of the features in this factor appear to correspond to a lack of clarity in the presentation of the content or task (23-inverse, 40, 63-inverse), as well as abstractness (27) and ambiguity (14). Curiously, feature 71-inverse (the lack of a specific type of implementation flaw in hint messages, which would make things very unclear) appears to point in the opposite direction – however, this implementation flaw was only common in a single rarely gamed lesson, so this result is probably a statistical artifact.

Feature 53-inverse appears to represent a different construct – interestingness (or the attempt to increase interestingness). The fact that feature 53 was associated with less gaming whereas more specific interest-increasing features (features 46-52) were not so strongly related may suggest that it is less important exactly how a problem scenario attempts to increase interest, than it is important that the problem scenario has some content in it that is not strictly mathematical.

Taken individually, two of the constructs in this factor were significantly (or marginally significantly) associated with gaming. Feature 53-inverse (text in the problem statement not directly related to the problem-solving task) was associated with significantly less gaming, r2 = 0.19, F(1,19) = 4.59, p = 0.04. Feature 40 (when it is not immediately apparent what icons in toolbar mean) was marginally significantly associated with more gaming, r2 = 0.15, F(1, 19)=3.52, p=0.08. The fact that other top features in the factor were not independently associated with gaming, while the factor as a whole was fairly strongly associated with gaming, suggests that gaming may occur primarily when more than one of these features are present.

Two features that were not present in the significant factor was statistically significantly associated with gaming: Feature 36, where the location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated, r2 = 0.20, F(1,19)=4.97, p=0.04. This feature, like many of those in the gaming-related factor, represents an unclear or confusing lesson. Also, Feature 79, whether or not the lesson was an equation solver unit, was statistically significantly better than chance, r2 = 0.30, F(1, 19)=8.55, p<0.01. Note, however, that although a lower amount of interesting text is generally associated with more gaming (Feature 53), equation-solver units (which have no text) have less gaming in general (Feature 79). This result may suggest that interest-increasing text is only beneficial (for reducing gaming) above a certain threshold -- alternatively, other aspects of the equation-solver units may have reduced gaming even though the lack of interesting-increasing text would generally be expected to increase it.

When the gaming-related factor, Feature 36, and Feature 79, were included in a model together, all remain statistically significant, and the combined model explains 56% of the variance in gaming (e.g. r2 = 0.55).

Five other features that were not strongly loaded in the significant factor were marginally associated with gaming. None of these other features is statistically significant in a model that already includes the gaming-related cluster and Feature 36. Due to the non-conclusiveness of the evidence relevant to these features, we will not discuss all of these features in detail, but will briefly mention one that has appeared in prior discussions of gaming. Lessons where a higher proportion of hint sequences told students what to do on the last hint (Feature 61) had marginally significantly more gaming, r2 = 0.14, F(1,19)=3.28, p=0.09. This result is unsurprising, as drilling through hints and typing in a bottom-out hint is one of the easiest and most frequently reported types of [[gaming the system]].

The off-task behavior model achieved similar predictive power, but was a much less complex model. None of the 6 factors were statistically significantly associated with gaming. Only one of the features was individually statistically significantly associated with off-task behavior: Feature 79, whether or not the lesson was an equation solver unit. Equation solver units had significantly less off-task behavior, just as they had significantly less gaming the system, and the effect was large in magnitude, r2 = 0.55, F(1, 21)=27.29, p<0.001, Bonferroni adjusted p<0.001.

To put this relationship into better context, we can look at the proportion of time students
spent off-task in equation-solver lessons as compared to other lessons. On average,
students spent 4.4% of their time off-task within the equation-solver lessons, much lower
than is generally seen in intelligent tutor classrooms or, for that matter, in traditional
classrooms. By contrast, students spent 14.1% of their time off-task within the
other lessons, a proportion of time-on-task which is much more in line with previous
observations. The difference in time spent per type of lesson is, as would be expected,
statistically significant, t(22)=4.48, p<0.001.

=== Connections to Other PSLC Studies===

This study inspired and led to the upcoming Year 6 study, [[Baker - Closing the Loop]].

===Annotated Bibliography===

===References===
Aleven, V., Koedinger, K.R. (2001) Investigations into Help Seeking and Learning with a Cognitive Tutor. In R. Luckin (Ed.), Papers of the AIED-2001 Workshop on Help Provision and Help Seeking in Interactive Learning Environments (2001) 47-58

Aleven, V., Stahl, E., Schworm, S., Fischer, F., Wallace, R. (2003) Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277-320.

Baker, R.S.J.d. (2007a) Is Gaming the System State-or-Trait? Educational Data Mining Through the Multi-Contextual Application of a Validated Behavioral Model. Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, 76-80. [http://www.joazeirodebaker.net/ryan/B2007B.pdf pdf]

Baker, R.S.J.d. (2007b) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S.J.d. (accepted) Differences Between Intelligent Tutor Lessons, and the Choice to Go Off-Task. To appear in Proceedings of EDM 2009.

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Aleven, V., de Carvalho, A., Raspat, J. (in press a) Educational Software Features that Encourage and Discourage "Gaming the System". To appear in Proceedings of AIED 2009.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.[http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]

Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (2008) Developing a Generalizable Detector of When Students Game the System. User Modeling and User-Adapted Interaction, 18, 3, 287-314. [http://www.joazeirodebaker.net/ryan/USER475.pdf pdf]

Baker, R.S., Corbett, A.T., Wagner, A.Z. (2006) Human Classification of Low-Fidelity Replays of Student Actions. Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems, 29-36. [http://www.joazeirodebaker.net/ryan/BCWFinal.pdf pdf]

Baker, R.S., Roll, I., Corbett, A.T., Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 57-64. [http://www.joazeirodebaker.net/ryan/BRCKAIED2005Final.pdf pdf]

Baker, R.S.J.d., Walonoski, J.A., Heffernan, N.T., Roll, I., Corbett, A.T., Koedinger, K.R. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224. [http://www.joazeirodebaker.net/ryan/BWHRKC-JILR-draft.pdf pdf]

Beck, J.E. (2006) Using Learning Decomposition to Analyze Student Fluency Development. Workshop on Educational Data Mining at the 8th International Conference on Intelligent Tutoring Systems, 21-28.

Brown, J.S., vanLehn, K. (1980) Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 379-426.

Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.

Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., Glaser, R. (1989) Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13, 145-182.

Corbett,A.T., & Anderson, J.R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.

Koedinger, K. R., & Anderson, J. R. (1998). Illustrating principled design: The early evolution of a cognitive tutor for algebra symbolization. Interactive Learning Environments, 5, 161-180.

Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]

Siegler, R.S. (2002) Microgenetic Studies of Self-Explanations. In N. Granott & J. Parziale (Eds.), Microdevelopment: Transition processes in development and learning, 31-58. New York: Cambridge University.

Walonoski, J.A., Heffernan, N.T. (2006) Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 382-391.

Baker - Closing the Loop

2009-12-05T20:55:06Z

Ryan: /* Abstract */ added link

== Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop” ==
=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker
|-
| '''Other Contributers''' ||

|-
| '''Study Start Date''' || Fall, 2009
|-
| '''Study End Date''' ||
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Middle-School Mathematics
|-
| '''Number of Students''' ||
|-
| '''Total Participant Hours''' ||
|-
| '''DataShop''' ||
|}

=== Abstract ===

This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC ([[Baker_Choices_in_LE_Space]]), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1) was created, and used to study why some tutor lessons are gamed more than others (in the Algebra tutor). The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed, and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

=== Background & Significance ===

=== Glossary ===

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

=== Hypotheses ===

;H1
:

;H2
:

;H3
:

;H4
:

=== Independent Variables ===
=== Dependent Variables ===

=== Planned Experiments ===

=== Explanation ===
=== Further Information ===
==== Connections ====
==== Annotated Bibliography ====
==== References ====
==== Future Plans ====

Gaming the system

2009-12-05T20:51:02Z

Ryan: /* PSLC Studies Involving Gaming the System */ added new projects

Baker et al (2006) defines gaming the system as "Attempting to succeed in an interactive learning environment by exploiting properties of the system rather than by learning the material".

Within intelligent tutoring systems such as Cognitive Tutors, this is usually done by systematic guessing, where a student systematically tries a set of possible answers (example: 1, 2, 3, 4, 5, ... 38) or hint abuse, drilling through hints at high speed to obtain the answer (Aleven & Koedinger, 2000).

Gaming has been observed in other types of learning environments as well, including educational games (Miller, Lehman, & Koedinger, 1999; Magnussen & Misfeldt, 2004; Rodrigo et al, 2007), simulation environments (Rodrigo et al, 2007), and graded-participation newsgroups (Cheng & Vassileva, 2005).

It has been repeatedly shown that students who game the system have poorer learning than non-gaming students with comparable pre-test scores (Baker et al, 2004, 2006; Walonoski & Heffernan, 2006a). (One exception is when students drill through hints, and then self-explain them -- Shih et al, 2008; another exception is when students game time-consuming material they already know -- Baker, Corbett & Koedinger, 2004).

Gaming the System has been shown to be associated with the affective experiences of boredom and confusion (Rodrigo et al, 2007) -- in specific, a student who experiences either of these two affective states is significantly more likely to be gaming the system shortly afterwards. Frustration, though previously found to be associated with gaming (Baker et al, 2008), appears to co-occur with gaming behavior rather than preceding it.

A variety of stable or semi-stable student characteristics have been studied in relation to gaming the system (e.g. Arroyo & Woolf, 2005; Baker et al, 2008; Beal, Qu, & Lee, 2009); however, these characteristics have generally been found to have weak correlations with gaming, at best. Some characteristics found to be significantly associated with gaming include negative attitudes towards computers, the learning software, and mathematics. Performance goals and anxiety have been repeatedly found to have no correlation to gaming (Baker et al, 2008).

Recent results from PSLC project [[Baker_Choices_in_LE_Space | How Content and Interface Features Influence Student Choices Within the Learning Space]] indicate that differences between tutor lessons explain much more of the variance in how much students choose to game, than individual differences between students. This finding was obtained through an ANOVA conducted at each of these two levels, and was replicated in both the middle school Cognitive Tutor (precursor to Bridge to Algebra) (Baker, 2007), and the Algebra Cognitive Tutor (paper in preparation).

Further data mining analysis (paper in preparation) using the [[CTLVS | Cognitive Tutor Lesson Variation Space (CTLVS)]] showed that students game the system more on lessons which have features which are likely to increase student confusion (including hints which do not lead any students to better performance, reference to abstract principles in hints, whether the toolbar is unclear, and the same number being used for multiple constructs) and boredom (including time-consuming problem steps and the lack of interest-increasing text in problem statements). These results conform well to the previous evidence on which affective states are associated with gaming.

[[Scooter the Tutor]] is a software agent who responds to gaming the system with emotional expressions and supplementary exercises (Baker et al, 2006). Scooter was associated with significantly reduced gaming, and significantly improved learning for gaming students (specifically those who received supplementary exercises). Scooter was built on top of the gaming detector, software validated to automatically detect gaming in running Cognitive Tutors (Baker et al, 2008).

=== PSLC Studies Involving Gaming the System ===

*[[Baker_Choices_in_LE_Space | How Content and Interface Features Influence Student Choices Within the Learning Space (Baker, Corbett, Koedinger, & Rodrigo)]]
* [[Baker - Closing the Loop]]
* [[Baker - Building Generalizable Fine-grained Detectors]]

=== See Also ===
Ryan Baker's [http://www.joazeirodebaker.net/ryan/gaming.html webpage on gaming the system].

=== Bibliography ===
* Aleven, V., Koedinger, K.R. (2000)Limitations of Student Control: Do Students Know When They Need Help? Proceedings of the 5th International Conference on Intelligent Tutoring Systems, 292-303.
* Baker, R.S.J.d. (2007) Is Gaming the System State-or-Trait? Educational Data Mining Through the Multi-Contextual Application of a Validated Behavioral Model. Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, 76-80. [http://www.cs.cmu.edu/~rsbaker/B2007B.pdf pdf]
* Baker, R.S., Corbett, A.T., Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 531-540. [http://www.cs.cmu.edu/~rsbaker/BCK2004MLFinal.pdf pdf]
* Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (2008) Developing a Generalizable Detector of When Students Game the System User Modeling and User-Adapted Interaction, 18, 3, 287-314. [http://www.joazeirodebaker.net/ryan/USER475.pdf pdf]
* Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S. E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006). Adapting to When Students Game an Intelligent Tutoring System. 8th International Conference on Intelligent Tutoring Systems, 392-401. [http://www.joazeirodebaker.net/ryan/Baker175.pdf pdf]
* Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System”. ACM CHI 2004: Computer-Human Interaction, 383-390. [http://www.joazeirodebaker.net/ryan/p383-baker-rev.pdf pdf]
* Baker, R.S.J.d., Walonoski, J.A., Heffernan, N.T., Roll, I., Corbett, A.T., Koedinger, K.R. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224. [http://www.joazeirodebaker.net/ryan/BWHRKC-JILR-draft.pdf pdf]
* Beal, C. R., Qu, L., & Lee, H. (2009). Mathematics motivation and achievement as predictors of high school students' guessing and help-seeking with instructional software. Journal of Computer Assisted Learning.
* Cheng, R., Vassileva, J. (2005) Adaptive Reward Mechanism for Sustainable Online Learning Community. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 152-159.
* Magnussen, R., Misfeldt, M. (2004) Player Transformation of Educational Multiplayer Games. Proceedings of Other Players. Available at [http://www.itu.dk/op/proceedings.htm http://www.itu.dk/op/proceedings.htm]
* Miller, C.S., Lehman, J.F., Koedinger, K.R. (1999) Goals and learning in microworlds - An exploration. Cognitive Science, 23 (3), 305-336.
* Murray, R.C., vanLehn, K. (2005) Effects of Dissuading Unnecessary Help Requests While Providing Proactive Help. Proceedings of the 12th International Conference on Artificial Intelligence in Education, 887-889.
* Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152.
[http://www.joazeirodebaker.net/ryan/RodrigoBakeretal2006Final.pdf pdf]
* Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126. [http://www.educationaldatamining.org/EDM2008/uploads/proc/12_Shih_35.pdf pdf]
* Walonoski, J.A., Heffernan, N.T. (2006a) Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 382-391.
* Walonoski, J.A., Heffernan, N.T. (2006b) Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 722-724.

[[Category:Glossary]]
[[Category:Interactive Communication]]
[[Category:Help Tutor]]

Baker - Closing the Loop

2009-12-05T20:49:50Z

Ryan: /* Glossary */

Baker - Closing the Loop

2009-12-05T20:49:26Z

Ryan: template + title

Baker - Building Generalizable Fine-grained Detectors

2009-12-05T20:46:16Z

Ryan:

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributers''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, Chinese
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, Chinese/FaCT, and the Chemistry Virtual Lab. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data, and questionnaire prompting (months 1-3) (in coordination with [[Nokes - Questionnaires]])

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7)

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)
* Randomized infrequent polling of student affect, motivation in popup windows
(“Which of these best describes how you’re feeling? [frustrated] [bored] [etc.]”) (in coordination with [[Nokes - Questionnaires]])

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al under, review)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Confusion (D'Mello et al, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010 and 2011, data will be collected in the Algebra, Geometry, Chemistry, and Chinese LearnLabs.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Baker - Building Generalizable Fine-grained Detectors

2009-12-05T20:44:14Z

Ryan:

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributers''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, Chinese
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, Chinese/FaCT, and the Chemistry Virtual Lab. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data, and questionnaire prompting (months 1-3) (in coordination with [[Nokes - Questionnaires]])

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7)

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)
* Randomized infrequent polling of student affect, motivation in popup windows
(“Which of these best describes how you’re feeling? [frustrated] [bored] [etc.]”) (in coordination with [[Nokes - Questionnaires]])

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al under, review)
* Boredom
* Confusion
* Frustration

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010 and 2011, data will be collected in the Algebra, Geometry, Chemistry, and Chinese LearnLabs.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===

Metacognition and Motivation

2009-12-05T20:43:29Z

Ryan:

The Metacognition and Motivation thrust has two broad goals, 1) to develop a better understanding of how metacognitive processes and motivation interact with learner factors to influence robust student learning outcomes and 2) to test whether and how student learning environments can leverage improved metacognition and motivation to increase the robustness of student learning. Our research will focus on a small number of metacognitive abilities (e.g., help seeking, self-explanation, interpreting peer feedback, and interpreting textual descriptions of domain principles), and a broader range of affective and motivational variables including: challenge perception, boredom, frustration, performance goals, and off-task behavior.

The Metacognition and Motivation thrust builds on the Coordinative Learning (CL) cluster, while bringing a significant shift of focus. The M&M thrust continues some of the work in the Coordinative Learning cluster that focused on the metacognitive aspects of coordinating multiple sources of information, such as studies on analogical comparison and self-explanation of examples by Nokes, and studies on diagrammatic self-explanation by Aleven and Butcher. It will also build on work done in other thrusts, for example the work done by Hausmann and VanLehn on scaffolding self-explanations in peer collaborative settings, the work on help seeking by Roll, Aleven, et al, and the work on studying [[gaming the system]] and [[Off-Task Behavior]] by Baker et al. In addition, the M&M thrust aims to place greater emphasis on issues of motivation within learning sciences research than has been done so far within the PSLC.

We have recruited three senior consultants who are helping to increase both the quality of the Metacognition and Motivation research and its visibility within broader communities of metacognition and motivation researchers. They are: Dr. Barry J. Zimmerman, a pre-eminent scholar in metacognition and motivation (e.g., Schunk & Zimmerman, 2008), Dr. Josh Aronson, a distinguished expert in stereotypes, self-esteem, motivation, and attitudes, and Dr. Andrew Elliott, a well-known expert in achievement motivation and social motivation.

We will pursue the following three broad research directions:
''Create and validate automated detectors for affect, motivation, and meta-cognition''. We will start by enhancing the LearnLab infrastructure with technology for automatically monitoring metacognitive and affective variables, at a much finer grain-size, over longer durations, and for more students, than has been previously possible.

Specifically, we combine observational and questionnaire data with student log data (e.g. response times, patterns of activity), to develop machine-learned models that monitor, in real-time and moment-by-moment, affective, motivational, and metacognitive variables in interactive learning environments. In particular, we will develop detectors of such constructs as [[gaming the system]], [[Off-Task Behavior]], [[help-seeking]], boredom, frustration, engaged concentration, perception of challenge, self-efficacy, and performance goals. Once created, these detectors will only draw on information that is available to the learning environment in the normal course of its operation (student log data at the keystroke level, timing, and semantic levels), without requiring extra sensors, enabling these detectors to be used in authentic, unmodified learning settings. The combination of machine learning with observational and questionnaire data has already been successful at detecting a limited set of relevant constructs such as perception of challenge (de Vicente & Pain, 2002), [[gaming the system]] (Baker et al., 2008), and [[Off-Task Behavior]] (Baker, 2007).

These detectors will be implemented in learning software used in multiple LearnLabs, and will enable PSLC researchers across thrusts to study motivation and metacognition as mediating variables when evaluating interventions aimed at enhancing the robustness of student learning.

''Evaluate interventions aimed at supporting different metacognitive abilities''. The PSLC’s particular strength in this area is studying metacognition within the context of interactive learning environments. A key question is whether such environments can be as effective in fostering or supporting metacognitive skills as they have been in improving domain-specific learning. A number of recent in vivo experiments have revealed significantly improved domain learning among students given metacognitive tutoring support for self-explanation (Aleven & Koedinger, 2002), error-correction (Mathan & Koedinger, 2005), video-based prompting and peer collaborative scaffolding of self-explanation (Craig et al. 2007, 2008, submitted; Hausmann & Vanlehn, 2007a, 2007b), and remedial instruction for material missed through meta-cognitive errors (Baker et al, 2006). We will build on this earlier work, with interventions that attempt to support four metacognitive abilities: help seeking, self-explanation, interpreting peer feedback, and interpreting textual descriptions of domain principles. An important goal in the Metacognition and Motivation thrust is to develop interactive learning environments that can help students internalize this support, solidifying and generalizing their metacognitive skills so they will no longer need external support in future learning situations, and can approach a new domain with a general set of skills that can facilitate learning.

To evaluate the effectiveness of the interventions aimed at enhancing metacognition, we will look not only at the normal indicators of robust domain-level learning (i.e., transfer and retention), but also (and in particular) at whether future learning is accelerated. As appropriate, the detectors for metacognitive behaviors developed under goal 1 will be used to evaluate whether the targeted metacognitive behavior is enhanced both while the intervention is in place (as a manipulation check) as well as in future learning situations (to evaluate its role as a potential cause of accelerated future learning).

''Evaluate interventions aimed at inducing positive affect and motivation to persist''. As a complement to investigating how cognitive learning principles can improve and support metacognitive ability, we will also study the effect of interventions aimed at enhancing motivation, as a way of uncovering relationships between motivation, affect, and metacognition in interactive learning environments. We will focus on two types of interventions. First, inspired by the motivational impact of computer games, we will (a) identify features of games that could be adopted for use in interactive learning environments, and (b) evaluate the effect of adding these features. Our initial investigations will focus on trivial choice, “boss problems,” (challenge problems – e.g. Siegler & Jenkins, 1981, designed to look like end-of-level bosses in video games) student control over challenge level, and rewards. In addition, we will evaluate the effect of putting the student in a care-taking role where they need to tutor a synthetic student. (The synthetic student will be driven by the PSLC’s SimStudent learning agent technology.) Second, we will evaluate social interventions aimed at enhancing motivation: peer pressure and comparison with peers, including competition with simulated students.

== Descendants ==

To create a new project page, enclose your project name in a double set of brackets. Details for a project format may be [[ Project_Page_Template_and_Creation_Instructions | found here.]]

*[[Nokes - Questionnaires]]
*[[Baker - Building Generalizable Fine-grained Detectors]]
*[[Roll - Inquiry]]
*[[Pavlik - Dificulty and Strategy]]
*[[Nokes - Dialectical Interaction and Robust Learning]]
*[[Math Game Elements]]
*[[Geometry Greatest Hits]]
*[[Aleven - Causal Argumentation Game]]

Metacognition and Motivation

2009-12-05T20:42:39Z

Ryan:

The Metacognition and Motivation thrust has two broad goals, 1) to develop a better understanding of how metacognitive processes and motivation interact with learner factors to influence robust student learning outcomes and 2) to test whether and how student learning environments can leverage improved metacognition and motivation to increase the robustness of student learning. Our research will focus on a small number of metacognitive abilities (e.g., help seeking, self-explanation, interpreting peer feedback, and interpreting textual descriptions of domain principles), and a broader range of affective and motivational variables including: challenge perception, boredom, frustration, performance goals, and off-task behavior.

The Metacognition and Motivation thrust builds on the Coordinative Learning (CL) cluster, while bringing a significant shift of focus. The M&M thrust continues some of the work in the Coordinative Learning cluster that focused on the metacognitive aspects of coordinating multiple sources of information, such as studies on analogical comparison and self-explanation of examples by Nokes, and studies on diagrammatic self-explanation by Aleven and Butcher. It will also build on work done in other thrusts, for example the work done by Hausmann and VanLehn on scaffolding self-explanations in peer collaborative settings, the work on help seeking by Roll, Aleven, et al, and the work on studying [[gaming the system]] and [[Off-Task Behavior]] by Baker et al. In addition, the M&M thrust aims to place greater emphasis on issues of motivation within learning sciences research than has been done so far within the PSLC.

We have recruited three senior consultants who are helping to increase both the quality of the Metacognition and Motivation research and its visibility within broader communities of metacognition and motivation researchers. They are: Dr. Barry J. Zimmerman, a pre-eminent scholar in metacognition and motivation (e.g., Schunk & Zimmerman, 2008), Dr. Josh Aronson, a distinguished expert in stereotypes, self-esteem, motivation, and attitudes, and Dr. Andrew Elliott, a well-known expert in achievement motivation and social motivation.

We will pursue the following three broad research directions:
''Create and validate automated detectors for affect, motivation, and meta-cognition''. We will start by enhancing the LearnLab infrastructure with technology for automatically monitoring metacognitive and affective variables, at a much finer grain-size, over longer durations, and for more students, than has been previously possible.

Specifically, we combine observational and questionnaire data with student log data (e.g. response times, patterns of activity), to develop machine-learned models that monitor, in real-time and moment-by-moment, affective, motivational, and metacognitive variables in interactive learning environments. In particular, we will develop detectors of such constructs as [[gaming the system]], [[off-task behavior]], [[help-seeking]], boredom, frustration, flow, perception of challenge, self-efficacy, and performance goals. Once created, these detectors will only draw on information that is available to the learning environment in the normal course of its operation (student log data at the keystroke level, timing, and semantic levels), without requiring extra sensors, enabling these detectors to be used in authentic, unmodified learning settings. The combination of machine learning with observational and questionnaire data has already been successful at detecting a limited set of relevant constructs such as perception of challenge (de Vicente & Pain, 2002), [[gaming the system]] (Baker et al., 2008), and [[off-task behavior]] (Baker, 2007).

These detectors will be implemented in learning software used in multiple LearnLabs, and will enable PSLC researchers across thrusts to study motivation and metacognition as mediating variables when evaluating interventions aimed at enhancing the robustness of student learning.

''Evaluate interventions aimed at supporting different metacognitive abilities''. The PSLC’s particular strength in this area is studying metacognition within the context of interactive learning environments. A key question is whether such environments can be as effective in fostering or supporting metacognitive skills as they have been in improving domain-specific learning. A number of recent in vivo experiments have revealed significantly improved domain learning among students given metacognitive tutoring support for self-explanation (Aleven & Koedinger, 2002), error-correction (Mathan & Koedinger, 2005), video-based prompting and peer collaborative scaffolding of self-explanation (Craig et al. 2007, 2008, submitted; Hausmann & Vanlehn, 2007a, 2007b), and remedial instruction for material missed through meta-cognitive errors (Baker et al, 2006). We will build on this earlier work, with interventions that attempt to support four metacognitive abilities: help seeking, self-explanation, interpreting peer feedback, and interpreting textual descriptions of domain principles. An important goal in the Metacognition and Motivation thrust is to develop interactive learning environments that can help students internalize this support, solidifying and generalizing their metacognitive skills so they will no longer need external support in future learning situations, and can approach a new domain with a general set of skills that can facilitate learning.

To evaluate the effectiveness of the interventions aimed at enhancing metacognition, we will look not only at the normal indicators of robust domain-level learning (i.e., transfer and retention), but also (and in particular) at whether future learning is accelerated. As appropriate, the detectors for metacognitive behaviors developed under goal 1 will be used to evaluate whether the targeted metacognitive behavior is enhanced both while the intervention is in place (as a manipulation check) as well as in future learning situations (to evaluate its role as a potential cause of accelerated future learning).

''Evaluate interventions aimed at inducing positive affect and motivation to persist''. As a complement to investigating how cognitive learning principles can improve and support metacognitive ability, we will also study the effect of interventions aimed at enhancing motivation, as a way of uncovering relationships between motivation, affect, and metacognition in interactive learning environments. We will focus on two types of interventions. First, inspired by the motivational impact of computer games, we will (a) identify features of games that could be adopted for use in interactive learning environments, and (b) evaluate the effect of adding these features. Our initial investigations will focus on trivial choice, “boss problems,” (challenge problems – e.g. Siegler & Jenkins, 1981, designed to look like end-of-level bosses in video games) student control over challenge level, and rewards. In addition, we will evaluate the effect of putting the student in a care-taking role where they need to tutor a synthetic student. (The synthetic student will be driven by the PSLC’s SimStudent learning agent technology.) Second, we will evaluate social interventions aimed at enhancing motivation: peer pressure and comparison with peers, including competition with simulated students.

== Descendants ==

To create a new project page, enclose your project name in a double set of brackets. Details for a project format may be [[ Project_Page_Template_and_Creation_Instructions | found here.]]

*[[Nokes - Questionnaires]]
*[[Baker - Building Generalizable Fine-grained Detectors]]
*[[Roll - Inquiry]]
*[[Pavlik - Dificulty and Strategy]]
*[[Nokes - Dialectical Interaction and Robust Learning]]
*[[Math Game Elements]]
*[[Geometry Greatest Hits]]
*[[Aleven - Causal Argumentation Game]]

Baker - Building Generalizable Fine-grained Detectors

2009-12-05T20:41:50Z

Ryan: refs added

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributers''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, Chinese
|-
| '''Number of Students''' || TBD
|-
| '''Total Participant Hours''' || TBD
|-
| '''DataShop''' || TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in DataShop, in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, Chinese/FaCT, and the Chemistry Virtual Lab. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[PSLC DataShop]] data, and questionnaire prompting (months 1-3) (in coordination with [[Nokes - Questionnaires]])

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7)

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)
* Randomized infrequent polling of student affect, motivation in popup windows
(“Which of these best describes how you’re feeling? [frustrated] [bored] [etc.]”) (in coordination with [[Nokes - Questionnaires]])

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al under, review)
* Boredom
* Confusion
* Frustration

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010 and 2011, data will be collected in the Algebra, Geometry, Chemistry, and Chinese LearnLabs.

=== Explanation ===
=== Further Information ===
=== Connections ===

[[Nokes - Questionnaires]]

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===