Application of SimStudent for Error Analysis - Revision history

Mbett: Reverted edits by Woolerystixmaker (Talk); changed back to last version by Nmatsuda

2011-08-29T13:30:39Z

Reverted edits by Woolerystixmaker (Talk); changed back to last version by Nmatsuda

← Older revision		Revision as of 13:30, 29 August 2011
Line 103:		Line 103:

	*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.		*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.

	~~[http://custom-essay-writing-service.org/index.php custom essay writing]~~

Woolerystixmaker at 09:24, 26 August 2011

2011-08-26T09:24:27Z

← Older revision		Revision as of 09:24, 26 August 2011
Line 103:		Line 103:

	*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.		*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.

			[http://custom-essay-writing-service.org/index.php custom essay writing]

Nmatsuda: /* Abstract */

2010-11-15T20:19:29Z

Abstract

← Older revision		Revision as of 20:19, 15 November 2010
Line 8:		Line 8:
	===Abstract===		===Abstract===

	The purpose of this project is to study how students ''fail to learn'' correct knowledge components when studying from examples and make (typical) errors by applying such incorrect knowledge components later when solving problems. We utilize a computational model of learning, called [http://www.cs.~~cmu.edu/~mazda/SimStudent~~ SimStudent] that learns cognitive skills inductively from examples either by passively reviewing worked-out examples or by actively engaged in tutored problem-solving.		The purpose of this project is to study how students ''fail to learn'' correct knowledge components when studying from examples and make (typical) errors by applying such incorrect knowledge components later when solving problems. We utilize a computational model of learning, called [http://www.SimStudent.org SimStudent] that learns cognitive skills inductively from examples either by passively reviewing worked-out examples or by actively engaged in tutored problem-solving.

	We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors.		We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors.

Nmatsuda: /* Overview of SimStudent */

2009-12-30T19:26:15Z

Overview of SimStudent

← Older revision		Revision as of 19:26, 30 December 2009
Line 16:		Line 16:
	===Overview of SimStudent===		===Overview of SimStudent===

	A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''~~operators~~'' and ''~~feature predicates~~'' as prior knowledge.		A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''feature predicates'' and ''operators'' as prior knowledge.

	Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4."		Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4."
Line 26:		Line 26:
	Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation.		Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation.

	Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples.		Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples.

	===Background and Significance===		===Background and Significance===

Nmatsuda: /* Abstract */

2009-12-30T18:36:16Z

Abstract

← Older revision		Revision as of 18:36, 30 December 2009
Line 8:		Line 8:
	===Abstract===		===Abstract===

	The purpose of this project is to study how students ''learn'' ~~errors~~ from examples. We ~~apply~~ a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively either ~~from~~ worked-out examples or by ~~being~~ tutored. ~~In this study, we use SimStudent to study how and when erroneous skills (the skills that produce errors when applied) would be learned.~~		The purpose of this project is to study how students ''fail to learn'' correct knowledge components when studying from examples and make (typical) errors by applying such incorrect knowledge components later when solving problems. We utilize a computational model of learning, called [http://www.cs.cmu.edu/~mazda/SimStudent SimStudent] that learns cognitive skills inductively from examples either by passively reviewing worked-out examples or by actively engaged in tutored problem-solving.

	We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors.		We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors.

	To test this hypothesis, we ~~give~~ SimStudent ~~different sets of~~ prior knowledge and analyze learning outcomes.		To test this hypothesis, we control SimStudent's prior knowledge to study how and when erroneous skills are learned by analyze learning outcomes (both the process of learning and the performance on the post-test).

	===Overview of SimStudent===		===Overview of SimStudent===

Nmatsuda: /* Learning Curve */

2009-05-15T03:50:03Z

Learning Curve

← Older revision		Revision as of 03:50, 15 May 2009
Line 73:		Line 73:

	Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.		Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.

			The Weak-PK and Strong-PK conditions had similar success rates on test problems after the first 8 training problems. After that, the performance of the two conditions began to diverge. On the final test after 20 training problems, the Strong-PK condition was 82% correct while the Weak-PK was 66%, a large and statistically significant difference (t = 4.00, p < .001).

	A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).		A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).

Nmatsuda: /* Error Prediction */

2009-05-15T03:49:13Z

Error Prediction

← Older revision		Revision as of 03:49, 15 May 2009
Line 83:		Line 83:

	Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations.		Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations.

			Surprisingly, the Weak PK condition did make as many as 22 human-like errors on the 11 test problems. On the other hand, the Strong PK condition hardly made human-like errors.

	[[Image:NM-Num-TN-Prediction.jpg]]		[[Image:NM-Num-TN-Prediction.jpg]]

Nmatsuda: /* Error Prediction */

2009-05-15T03:48:31Z

Error Prediction

← Older revision		Revision as of 03:48, 15 May 2009
Line 84:		Line 84:
	Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations.		Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations.

	[[Image:NM-Num-TN-~~prediction~~.jpg]]		[[Image:NM-Num-TN-Prediction.jpg]]

	Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.		Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.

Nmatsuda: /* Learning Curve */

2009-05-15T03:46:32Z

Learning Curve

← Older revision		Revision as of 03:46, 15 May 2009
Line 73:		Line 73:

	Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.		Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.

			A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).

	[[Image:NM-LearningCurve.jpg]]		[[Image:NM-LearningCurve.jpg]]

	Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.		Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.

			====Error Prediction====

			Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations.

			[[Image:NM-Num-TN-prediction.jpg]]

			Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.

	===Publications===		===Publications===

Nmatsuda: /* Findings */

2009-05-15T03:43:06Z

Findings

← Older revision		Revision as of 03:43, 15 May 2009
Line 69:		Line 69:

	===Findings===		===Findings===

			====Learning Curve====

	Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.		Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.