https://learnlab.org/research/wiki/api.php?action=feedcontributions&user=Nmatsuda&feedformat=atomLearnLab - User contributions [en]2024-03-28T16:47:55ZUser contributionsMediaWiki 1.31.12https://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=11275Application of SimStudent for Error Analysis2010-11-15T20:19:29Z<p>Nmatsuda: /* Abstract */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''fail to learn'' correct knowledge components when studying from examples and make (typical) errors by applying such incorrect knowledge components later when solving problems. We utilize a computational model of learning, called [http://www.SimStudent.org SimStudent] that learns cognitive skills inductively from examples either by passively reviewing worked-out examples or by actively engaged in tutored problem-solving. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we control SimStudent's prior knowledge to study how and when erroneous skills are learned by analyze learning outcomes (both the process of learning and the performance on the post-test).<br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''feature predicates'' and ''operators'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples.<br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
====Learning Curve====<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
The Weak-PK and Strong-PK conditions had similar success rates on test problems after the first 8 training problems. After that, the performance of the two conditions began to diverge. On the final test after 20 training problems, the Strong-PK condition was 82% correct while the Weak-PK was 66%, a large and statistically significant difference (t = 4.00, p < .001). <br />
<br />
A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
====Error Prediction====<br />
<br />
Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations. <br />
<br />
Surprisingly, the Weak PK condition did make as many as 22 human-like errors on the 11 test problems. On the other hand, the Strong PK condition hardly made human-like errors. <br />
<br />
[[Image:NM-Num-TN-Prediction.jpg]]<br />
<br />
Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=10409Application of SimStudent for Error Analysis2009-12-30T19:26:15Z<p>Nmatsuda: /* Overview of SimStudent */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''fail to learn'' correct knowledge components when studying from examples and make (typical) errors by applying such incorrect knowledge components later when solving problems. We utilize a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively from examples either by passively reviewing worked-out examples or by actively engaged in tutored problem-solving. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we control SimStudent's prior knowledge to study how and when erroneous skills are learned by analyze learning outcomes (both the process of learning and the performance on the post-test).<br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''feature predicates'' and ''operators'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples.<br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
====Learning Curve====<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
The Weak-PK and Strong-PK conditions had similar success rates on test problems after the first 8 training problems. After that, the performance of the two conditions began to diverge. On the final test after 20 training problems, the Strong-PK condition was 82% correct while the Weak-PK was 66%, a large and statistically significant difference (t = 4.00, p < .001). <br />
<br />
A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
====Error Prediction====<br />
<br />
Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations. <br />
<br />
Surprisingly, the Weak PK condition did make as many as 22 human-like errors on the 11 test problems. On the other hand, the Strong PK condition hardly made human-like errors. <br />
<br />
[[Image:NM-Num-TN-Prediction.jpg]]<br />
<br />
Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=10408Application of SimStudent for Error Analysis2009-12-30T18:36:16Z<p>Nmatsuda: /* Abstract */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''fail to learn'' correct knowledge components when studying from examples and make (typical) errors by applying such incorrect knowledge components later when solving problems. We utilize a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively from examples either by passively reviewing worked-out examples or by actively engaged in tutored problem-solving. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we control SimStudent's prior knowledge to study how and when erroneous skills are learned by analyze learning outcomes (both the process of learning and the performance on the post-test).<br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
====Learning Curve====<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
The Weak-PK and Strong-PK conditions had similar success rates on test problems after the first 8 training problems. After that, the performance of the two conditions began to diverge. On the final test after 20 training problems, the Strong-PK condition was 82% correct while the Weak-PK was 66%, a large and statistically significant difference (t = 4.00, p < .001). <br />
<br />
A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
====Error Prediction====<br />
<br />
Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations. <br />
<br />
Surprisingly, the Weak PK condition did make as many as 22 human-like errors on the 11 test problems. On the other hand, the Strong PK condition hardly made human-like errors. <br />
<br />
[[Image:NM-Num-TN-Prediction.jpg]]<br />
<br />
Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=10098Computational Modeling and Data Mining2009-11-20T23:27:00Z<p>Nmatsuda: /* Descendants */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Analysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers. <br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== Descendants ==<br />
<br />
*[[Gordon - Temporal learning for EDM]]<br />
*[[Koedinger - Discovery of Domain-Specific Cognitive Models]]<br />
*[[Koedinger - Toward a model of accelerated future learning]]<br />
*[[Baker - Building generalizable fine-grained detectors]]<br />
*[[Chi - Induction of Adaptive Pedagogical Tutorial Tactics]]<br />
*[[Baker - Closing the Loop]]<br />
*[[Pavlik - Generalizing the Assistance Formula]]<br />
*[[Mayer_and_McLaren_-_Social_Intelligence_And_Computer_Tutors | McLaren and Mayer - Social Intelligence and Learning from "polite" tutors]]<br />
*[[Matsuda - Application of SimStudent for Error Analysis]]<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.<br />
* McLaren, B.M., Lim, S., & Koedinger, K.R. (2008). When and How Often Should Worked Examples be Given to Students? New Results and a Summary of the Current State of Research. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 2176-2181). Austin, TX: Cognitive Science Society. <br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9364Application of SimStudent for Error Analysis2009-05-15T03:50:03Z<p>Nmatsuda: /* Learning Curve */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
====Learning Curve====<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
The Weak-PK and Strong-PK conditions had similar success rates on test problems after the first 8 training problems. After that, the performance of the two conditions began to diverge. On the final test after 20 training problems, the Strong-PK condition was 82% correct while the Weak-PK was 66%, a large and statistically significant difference (t = 4.00, p < .001). <br />
<br />
A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
====Error Prediction====<br />
<br />
Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations. <br />
<br />
Surprisingly, the Weak PK condition did make as many as 22 human-like errors on the 11 test problems. On the other hand, the Strong PK condition hardly made human-like errors. <br />
<br />
[[Image:NM-Num-TN-Prediction.jpg]]<br />
<br />
Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9363Application of SimStudent for Error Analysis2009-05-15T03:49:13Z<p>Nmatsuda: /* Error Prediction */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
====Learning Curve====<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
====Error Prediction====<br />
<br />
Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations. <br />
<br />
Surprisingly, the Weak PK condition did make as many as 22 human-like errors on the 11 test problems. On the other hand, the Strong PK condition hardly made human-like errors. <br />
<br />
[[Image:NM-Num-TN-Prediction.jpg]]<br />
<br />
Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9362Application of SimStudent for Error Analysis2009-05-15T03:48:31Z<p>Nmatsuda: /* Error Prediction */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
====Learning Curve====<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
====Error Prediction====<br />
<br />
Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations. <br />
<br />
[[Image:NM-Num-TN-Prediction.jpg]]<br />
<br />
Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=File:NM-Num-TN-Prediction.jpg&diff=9361File:NM-Num-TN-Prediction.jpg2009-05-15T03:48:12Z<p>Nmatsuda: </p>
<hr />
<div></div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9360Application of SimStudent for Error Analysis2009-05-15T03:46:32Z<p>Nmatsuda: /* Learning Curve */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
====Learning Curve====<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
====Error Prediction====<br />
<br />
Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations. <br />
<br />
[[Image:NM-Num-TN-prediction.jpg]]<br />
<br />
Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9359Application of SimStudent for Error Analysis2009-05-15T03:43:06Z<p>Nmatsuda: /* Findings */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
====Learning Curve====<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9358Application of SimStudent for Error Analysis2009-05-15T03:42:35Z<p>Nmatsuda: /* Findings */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9357Application of SimStudent for Error Analysis2009-05-15T03:41:47Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.<br />
[[Image:NM-LearningCurve.jpg]]<br />
<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=File:NM-LearningCurve.jpg&diff=9356File:NM-LearningCurve.jpg2009-05-15T03:40:18Z<p>Nmatsuda: </p>
<hr />
<div></div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9355Application of SimStudent for Error Analysis2009-05-15T03:32:08Z<p>Nmatsuda: /* Dependent Variables */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.<br />
<br />
===Findings===<br />
<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9354Application of SimStudent for Error Analysis2009-05-15T02:28:14Z<p>Nmatsuda: /* Dependent Variables */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:<br />
<br />
: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.<br />
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).<br />
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).<br />
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.<br />
<br />
'''Error rate''':<br />
<br />
===Findings===<br />
<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9353Application of SimStudent for Error Analysis2009-05-15T02:24:31Z<p>Nmatsuda: /* Dependent Variables */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application Ri (i = 1, …, n) is coded as follows:<br />
<br />
True Positive: Ri yields the same step as S, and S is a correct step.<br />
False Positive: Ri yields a correct step that is not same as S (S may be incorrect).<br />
False Negative: Ri yields an incorrect step that is not same as S (S may be correct).<br />
True Negative: Ri yields the same step as S and S is an incorrect step.<br />
<br />
'''Error rate''':<br />
<br />
===Findings===<br />
<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9352Application of SimStudent for Error Analysis2009-05-15T02:22:15Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step. <br />
<br />
'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step S performed by a human student at an intermediate state �, SimStudent is asked to compute a conflict set on �. Rule application Ri (i = 1, …, n) is coded as follows:<br />
<br />
True Positive: Ri yields the same step as S, and S is a correct step.<br />
False Positive: Ri yields a correct step that is not same as S (S may be incorrect).<br />
False �egative: Ri yields an incorrect step that is not same as S (S may be correct).<br />
True �egative: Ri yields the same step as S and S is an incorrect step.<br />
<br />
'''Error rate''': <br />
<br />
===Findings===<br />
<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9331Application of SimStudent for Error Analysis2009-05-14T21:45:53Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Human Students Error Analysis===<br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors. <br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
'''Error prediction''': <br />
<br />
'''Error rate''': <br />
<br />
===Findings===<br />
<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9330Application of SimStudent for Error Analysis2009-05-14T21:31:36Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator <br />
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C. <br />
<br />
===Research Question===<br />
<br />
How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make? <br />
<br />
===Hypothesis===<br />
<br />
<br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9329Application of SimStudent for Error Analysis2009-05-14T21:16:44Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent=== <br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.<br />
<br />
We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced. <br />
<br />
We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions. <br />
<br />
===Research Question===<br />
<br />
<br />
<br />
===Hypothesis===<br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9328Application of SimStudent for Error Analysis2009-05-14T20:32:01Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Overview of SimStudent=== <br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation. <br />
<br />
Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples. <br />
<br />
===Background and Significance===<br />
<br />
There are a number of studies <br />
<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9317Application of SimStudent for Error Analysis2009-05-14T17:35:23Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Background and Significance===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill. <br />
<br />
Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. For the former case, <br />
<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9290Application of SimStudent for Error Analysis2009-05-14T16:48:10Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Background and Significance===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4." <br />
<br />
To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill. <br />
<br />
<br />
<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9284Application of SimStudent for Error Analysis2009-05-14T16:26:25Z<p>Nmatsuda: /* Towards a theory of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
===Personnel===<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either from worked-out examples or by being tutored. In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we give SimStudent different sets of prior knowledge and analyze learning outcomes. <br />
<br />
===Background and Significance===<br />
<br />
A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''operators'' and ''feature predicates'' as prior knowledge. <br />
<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.<br />
<br />
*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.<br />
<br />
*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9135Application of SimStudent for Error Analysis2009-05-08T14:43:54Z<p>Nmatsuda: /* An application of a computational model of learning as a model of learning errors */</p>
<hr />
<div>==Towards a theory of learning errors==<br />
<br />
Noboru Matsuda, William W. Cohen, & Kenneth R. Koedinger <br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively from examples. In this study, we use SimStudent to mode a process of learning to study how and when erroneous skills (the skills that produce errors when applied) would be learned with the prior knowledge as a control variable. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we provide SimStudent different set of prior knowledge and measure learning outcomes. <br />
<br />
<br />
===Background and Significance===<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9122Application of SimStudent for Error Analysis2009-05-07T20:16:07Z<p>Nmatsuda: /* An application of a computational model of learning as a model of learning errors */</p>
<hr />
<div>==An application of a computational model of learning as a model of learning errors==<br />
<br />
Noboru Matsuda, William W. Cohen, & Kenneth R. Koedinger <br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively from examples. In this study, we use SimStudent to mode a process of learning to study how and when erroneous skills (the skills that produce errors when applied) would be learned with the prior knowledge as a control variable. <br />
<br />
We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors. <br />
<br />
To test this hypothesis, we provide SimStudent different set of prior knowledge and measure learning outcomes. <br />
<br />
<br />
===Background and Significance===<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Study Variables===<br />
<br />
====Independent Variable====<br />
<br />
'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent. <br />
<br />
====Dependent Variables====<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9121Application of SimStudent for Error Analysis2009-05-07T19:03:09Z<p>Nmatsuda: /* Application of SimStudent for Error Analysis */</p>
<hr />
<div>==An application of a computational model of learning as a model of learning errors==<br />
<br />
Noboru Matsuda, William W. Cohen, & Kenneth R. Koedinger <br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. We apply a computational model of learning, called SimStudent <br />
<br />
===Background and Significance===<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9111Application of SimStudent for Error Analysis2009-05-06T23:27:15Z<p>Nmatsuda: /* Application of SimStudent for Error Analysis */</p>
<hr />
<div>==Application of SimStudent for Error Analysis==<br />
<br />
Noboru Matsuda, William W. Cohen, & Kenneth R. Koedinger <br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors from examples. <br />
<br />
===Background and Significance===<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Findings===<br />
<br />
====Impact of having "weak" prior knowledge in learning errors====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===<br />
<br />
*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9110Application of SimStudent for Error Analysis2009-05-06T23:24:59Z<p>Nmatsuda: /* Application of SimStudent for Error Analysis */</p>
<hr />
<div>==Application of SimStudent for Error Analysis==<br />
<br />
Noboru Matsuda, William W. Cohen, & Kenneth R. Koedinger <br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
===Abstract===<br />
<br />
The purpose of this project is to study how students ''learn'' errors <math>2x+1</math><br />
<br />
===Background and Significance===<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Findings===<br />
<br />
=====Impact of having "weak" prior knowledge in learning errors=====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9109Application of SimStudent for Error Analysis2009-05-06T23:00:48Z<p>Nmatsuda: </p>
<hr />
<div>==Application of SimStudent for Error Analysis==<br />
<br />
Noboru Matsuda, William W. Cohen, & Kenneth R. Koedinger <br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
<br />
===Abstract===<br />
<br />
===Background and Significance===<br />
<br />
===Research Question===<br />
<br />
===Hypothesis===<br />
<br />
===Findings===<br />
<br />
=====Impact of having "weak" prior knowledge in learning errors=====<br />
<br />
===Publications===<br />
<br />
*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.<br />
<br />
===References===</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=9108Computational Modeling and Data Mining2009-05-06T23:00:03Z<p>Nmatsuda: /* Developing Models of ''Domain-General'' Learning and Motivational Processes */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Analysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers. <br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=9107Computational Modeling and Data Mining2009-05-06T22:59:22Z<p>Nmatsuda: /* Developing Models of ''Domain-General'' Learning and Motivational Processes */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent project] both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Analysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers. <br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=9106Computational Modeling and Data Mining2009-05-06T22:58:59Z<p>Nmatsuda: /* Developing Models of ''Domain-General'' Learning and Motivational Processes */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the [http://www.cs.cmu.edu/~mazda/SimStudent|SimStudent project] both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Analysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers. <br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9105Application of SimStudent for Error Analysis2009-05-06T22:52:40Z<p>Nmatsuda: </p>
<hr />
<div>==Application of SimStudent for Error Analysis==<br />
<br />
Noboru Matsuda, William W. Cohen, & Kenneth R. Koedinger <br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger<br />
<br />
<br />
===Abstract===<br />
<br />
===Research Question===<br />
<br />
===Background and Significance===</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Application_of_SimStudent_for_Error_Analysis&diff=9104Application of SimStudent for Error Analysis2009-05-06T22:41:36Z<p>Nmatsuda: New page: ==Application of SimStudent for Error Analysis== Noboru Matsuda, William W. Cohen, Kenneth R. Koedinger *PI: Noboru Matsuda *Key Faculty: William W. Cohen, Kenneth R. Koedinger</p>
<hr />
<div>==Application of SimStudent for Error Analysis==<br />
<br />
Noboru Matsuda, William W. Cohen, Kenneth R. Koedinger<br />
<br />
*PI: Noboru Matsuda<br />
*Key Faculty: William W. Cohen, Kenneth R. Koedinger</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=9100Computational Modeling and Data Mining2009-05-06T22:29:23Z<p>Nmatsuda: /* Developing Models of ''Domain-General'' Learning and Motivational Processes */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the SimStudent project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Analysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers. <br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=9099Computational Modeling and Data Mining2009-05-06T22:28:59Z<p>Nmatsuda: /* Developing Models of ''Domain-General'' Learning and Motivational Processes */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the SimStudent project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009; see [[Application of SimStudent for Error Aalysis]]). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers. <br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=8818Computational Modeling and Data Mining2009-02-11T06:38:53Z<p>Nmatsuda: /* References */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the SimStudent project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers. <br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2008). Why tutored problem solving may be better than example study: Theoretical implications from a simulated-student study. In B. P. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 111-121). Heidelberg, Berlin: Springer.<br />
* Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007). Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the international conference on User Modeling (LNAI 4511) (pp. 107-116). Berlin, Heidelberg: Springer.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=8817Computational Modeling and Data Mining2009-02-11T06:37:05Z<p>Nmatsuda: /* Developing Models of ''Domain-General'' Learning and Motivational Processes */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the SimStudent project both as a theory-building tool and as an instruction-informing tool (Matsuda et al., 2008). We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop (Matsuda et al., 2009). While psychological and neuroscientific models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors . Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers. <br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=8686Computational Modeling and Data Mining2008-12-08T18:16:01Z<p>Nmatsuda: /* Developing Predictive ''Engineering Models'' to Inform Instructional Event Design */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the SimStudent project both as a theory-building tool and as an instruction-informing tool. We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop. While psychological and neuroscientificce models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors. Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers.<br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsudahttps://learnlab.org/wiki/index.php?title=Computational_Modeling_and_Data_Mining&diff=8685Computational Modeling and Data Mining2008-12-08T18:12:21Z<p>Nmatsuda: /* Developing Predictive ''Engineering Models'' to Inform Instructional Event Design */</p>
<hr />
<div>==Introduction==<br />
One of the greatest impacts of technology on 21st century education will be the scientific advances made possible by mining the vast explosion of learning data that is coming from educational technologies. The Computational Modeling and Data Mining (CMDM) Thrust is pursuing the scientific goal of using such data to advance precise, computational theories of how students learn academic content. We will accomplish this by drawing on and expanding the enabling technologies we have already built for collecting, storing, and managing large-scale educational data sets. For example, [http://www.learnlab.org/technologies/datashop/index.php DataShop] will grow to include larger and richer datasets coming not only from our LearnLab courses but also from thousands of schools using the Cognitive Tutor courses and from additional contexts where we can collect student dialogue data, measures of motivation and affect, and layered assessments of both student knowledge and metacognitive competencies. This growth in the amount, scope, and richness of learning data will make the [http://www.learnlab.org/technologies/datashop/index.php DataShop] an even more fertile cyber-infrastructure resource for learning science researchers to use. But to realize the full potential of that resource – to make new discoveries about the nature of student learning – researchers need new and powerful knowledge discovery tools – innovations that will occur within the CMDM Thrust.<br />
<br />
The CMDM Thrust will pursue three related areas: 1) domain-specific models of student knowledge representation and acquisition, 2) domain-general models of [[Metacognition and Motivation|metacognitive, motivational]], and [[Social_and_Communicative_Factors_in_Learning|social processes]] as they impact student learning, and 3) predictive engineering models and methods that enable the design of large-impact instructional interventions.<br />
<br />
== Developing Better Cognitive Models of ''Domain-Specific Content''==<br />
Understanding and engineering better human learning of complex academic topics is dependent upon accurate and usable models of the domains students are learning that result from [[cognitive task analysis]]. However, domain modeling has been a continual challenge, as student knowledge is not directly observable and its structure is often hidden by our “expert blind spots” ([[User:Koedinger|Koedinger]] & Nathan, 2004; Nathan & Koedinger, 2000). Key research questions are: a) Can the discovery of a domain’s knowledge structure be automated? b) Do [[knowledge component]] models provide a precise and predictive theory of [[transfer]] of learning? c) Can we integrate separate methods for modeling memory, learning, transfer, and guessing/slipping, to optimize models of student knowledge, and in turn optimize students' effective time on task?<br />
<br />
One of the planned projects for Year 5 will build on our promising past results, obtained with the Cen, Koedinger, and Junker (2006) Learning Factor Analysis (LFA) algorithms. Specifically, we will, by broadening the generalizability of this domain-modeling approach, incorporating new knowledge-discovery methods, and increasing the level of automation of knowledge analysis so as to engage more researchers in applying this technique to even more content domains. To more fully automate the discovery of knowledge components, Pavlik will use Partially Ordered Knowledge Structures (POKS) (cf. Desmarais, et al., 1995) to build more complete and accurate representations of map the given domain and to capture the prerequisite relationships between hypothesized knowledge components and their predictions of performance. The models that this work produces will become the input to algorithms that can optimize for each student the amount of practice and ideal sequencing of instructional events for acquiring each knowledge component. These approaches will be applied to tutors across domains, including math, science, and language (particularly for English vocabulary and article learning domains). A related project will investigate the impact of combining LFA model refinement with improved moment-by-moment knowledge modeling, using a probabilistic model that uses student interaction data to estimate whether a student’s correct answer or error informs us about their knowledge or simply represents a guess or slip (Baker, Corbett & Aleven, 2008). In addition to clear applied benefits, these projects will advance a more precise science of reasoning and learning as it occurs in academic settings.<br />
<br />
==Developing Models of ''Domain-General'' Learning and Motivational Processes==<br />
Our work toward developing high-fidelity models of student learning has involved capturing, quantifying, and modeling domain-general mechanisms that impact students’ learning and the robustness of that learning. In the first four years of the PSLC, our models have moved beyond addressing domain-specific cognition (e.g., the cognitive models behind the intelligent tutors for Physics, Algebra, and Geometry) to capture metacognitive aspects of learning (e.g., Aleven et al.’s, 2006, detailed model of help-seeking behavior), general mechanisms of learning (Matsuda et al., 2007) and motivational and affective constructs such as students’ off-task behavior (Baker, 2007), and whether a student is “gaming the system” (Baker et al., 2008; shown to be associated with boredom and confusion in Rodrigo et al, 2007). <br />
<br />
A key Year 5 effort will extend the SimStudent project both as a theory-building tool and as an instruction-informing tool. We will use SimStudent to make predictions about the nature of students’ generalization errors and the effects of prior knowledge on students’ learning and transfer, testing these predictions using human-learning data in DataShop. While psychological and neuroscientificce models typically produce only reaction time predictions, these models will predict specific errors and forecast the pattern of reduction in those errors. Developing a system that integrates domain-general processes to produce human-like errors in inference, calculation, generalization, and the use of feedback/help/instructions would be both a major theoretical breakthrough, and an extremely useful tool for other researchers.<br />
<br />
Looking forward to the renewal period, an important project will be to develop machine-learned models of student behaviors at a range of time scales, from momentary affective states like boredom and frustration (cf. Kapoor, Burleson, & Picard, 2007) to longer-term motivational and metacognitive constructs such as performance vs. learning orientation and self-regulated learning (Azevedo & Cromley, 2004; Elliott & Dweck, 1988; Pintrich, 2000; Winne & Hadwin, 1998). We will expand prior PSLC work by Baker and colleagues (Rodrigo et al, 2007, 2008; Baker et al, 2008) to explore causal connections between these models and existing models of motivation-related behaviors such as gaming the system and off-task behavior. We will pursue models of differences in cognitive, affective, social, and motivational factors as they relate to classroom culture, schools, and teachers. These proposed models would be, to our knowledge, the first systematic investigations of school-level effects factors affectingon fine-grained states of student learning.<br />
<br />
==Developing Predictive ''Engineering Models'' to Inform Instructional Event Design==<br />
A fundamental theoretical problem for the sciences of learning and instruction is what we have called “the [[assistance dilemma|Assistance Dilemma]]”: optimizing the amount and timing of instruction so that it is neither too little nor too much, and neither too early nor too late (Koedinger & Aleven, 2007; Koedinger, 2008; Koedinger, Pavlik, McLaren, & Aleven, 2008). Two theoretical advances are necessary before we can resolve these broad questions. First, we need a clear delineation of the multiple possible dimensions of instructional assistance (e.g., worked examples, feedback, on-demand hints, self-explanation prompts, or optimally-spaced practice trials). We broadly define assistance to include not only direct verbal instruction, but also instructional scaffolds that prompt student thinking or action as well as implicit affordances or difficulties in the learning environment. Second, we need precise, predictive models of when increasing assistance (reducing difficulties) or decreasing assistance (increasing difficulties) is best for optimal robust learning. Existing theoretical work on this topic – like [[cognitive load]] theory (e.g., Sweller, 1994; van Merrienboer & Sweller, 2005), desirable difficulties (Bjork, 1994), and cognitive apprenticeship (Collins, Brown, & Newman, 1989) -- have not reached the stage of precise computational modeling that can be used to make a priori predictions about optimal levels of assistance. <br />
<br />
We will use DataShop log data to make progress on the Assistance Dilemma by targeting dimensions of assistance one at a time and creating parameterized mathematical models that predict the optimal level of assistance to enhance robust learning (cf. Koedinger et al., 2008). Such a mathematical model has been achieved for the practice-interval dimension (changing the amount of time between practice trials), and progress is being made on the example-problem dimension (changing the ratio of examples to problems). These models generate the inverted-U shaped function curve characteristic of the Assistance Dilemma as a function of particular parameter values that describe the instructional context. These models are created and refined using student learning data from DataShop. We hypothesize that this form approach will work for other dimensions of assistance. These models will address the limitations of current theory indicated above by generating ''a priori'' predictions of what forms of assistance or difficulty will enhance learning. Further, these models will provide the basis for on-line algorithms that adapt to individual student differences and changes over time, optimizing the assistance provided to each student for each knowledge component at each time in their learning trajectory.<br />
<br />
[[Application of SimStudent for Error Aalysis]]<br />
<br />
== [[CMDM Meetings]] ==<br />
<br />
== References ==<br />
* Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology, 96(3), 523-535.<br />
* Bjork, R.A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.) Metacognition: Knowing about knowing. (pp.185-205). Cambridge, MA: MIT Press.<br />
* Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick. Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.<br />
* Desmarais, M., Maluf, A., Liu, J. (1995) User-expertise modeling with empirically derived probabilistic implication networks. User Modeling and User-Adapted Interaction, 5 (3-4), 283-315.<br />
* [[User:Koedinger|Koedinger]], K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264.<br />
* Koedinger, K. R., Pavlik Jr., P. I., McLaren, B. M., & Aleven, V. (2008). Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction. In B.C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society. (pp.). Austin, TX: Cognitive Science Society.<br />
* Nathan, M. J. & Koedinger, K.R. (2000). Teachers' and researchers' beliefs of early algebra development. Journal for Research in Mathematics Education, 31 (2), 168-190<br />
* Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295–312.<br />
* [http://www.ou.nl/eCache/DEF/7/332.html Van Merriënboer, J.J.G.], & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(1), 147-177.</div>Nmatsuda