The Help Tutor Roll Aleven McLaren - Revision history

Alida: /* Background and Significance */

2010-12-16T00:11:17Z

Background and Significance

← Older revision		Revision as of 00:11, 16 December 2010
Line 34:		Line 34:

	Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources\|help facilities].		Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources\|help facilities].
	However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation\|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.		However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[Gaming the system\|game the system]]" or do not [[self-explanation\|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

	==== Shallow help seeking [[knowledge component]]s ====		==== Shallow help seeking [[knowledge component]]s ====

Alida: /* Meta-data */

2010-11-02T17:25:49Z

Meta-data

← Older revision		Revision as of 17:25, 2 November 2010
Line 6:		Line 6:
	PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger		PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

	Other ~~Contributers~~: EJ Ryu (programmer)		Other Contributors: EJ Ryu (programmer)
	{\| border="1"		{\| border="1"
	! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?		! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?

Idoroll: /* Evaluation of goal 4: Improve future metacognitive behavior */

2009-05-21T04:23:19Z

Evaluation of goal 4: Improve future metacognitive behavior

← Older revision		Revision as of 04:23, 21 May 2009
Line 197:		Line 197:
	==== Evaluation of goal 4: Improve future metacognitive behavior ====		==== Evaluation of goal 4: Improve future metacognitive behavior ====

	~~Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably~~, ~~help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond~~ the ~~scope~~ of ~~the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the~~ study). We ~~hypothesize that since the Help Tutor feedback appeared~~ in ~~two different areas of geometry learning, students could more easily acquire~~ the ~~domain-independent help-seeking skills and thus transfer them better to~~ the ~~other subject matter areas addressed between~~ and ~~after~~ the ~~two~~ in ~~which the Help Tutor was used~~.		As mentioned earlier, the two units of study 2 were spread apart by one month. We collected data from students' behavior in the months between the two units and following the study. During these months students repeated previous material in preparation for statewide exams.
	~~The addition~~ of the help-seeking ~~instruction and self-assessment episodes in Study 2 led~~ to ~~improvements in students’ conceptual~~ help seeking ~~knowledge,~~ time ~~to read hints~~, ~~and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask~~ for hints ~~rather than committing more errors provide behavioral evidence that self~~-~~assessment instruction provided~~ by ~~both~~ the ~~Self-Assessment tutor and by certain messages in~~ the ~~Help Tutor appear to lead students to be more aware of their need for help~~, ~~reinforcing the causal link between self-assessment and strategic help seeking~~ (~~Tobias & Everson~~, ~~2002~~).		As seen in the table below, the main effects of the help-seeking environment persisted even once students moved on to working with the native Cognitive Tutor! Overall, students who received help-seeking support during the study took more time for their actions following the study, especially for reading hints - their hint reading time before asking for additional hint is longer by almost one effect size in the month following the study (12 vs. 8 seconds, t(44)=3.0, p<.01). Also, students who were in the Help condition did not drill down through the hints as often, though this effect is only marginal significant (average hint level: 2.2 vs. 2.6, t(51)=1.9, p=.06). These effects are more consistently significant after both units, suggesting that having the study stretched across two units indeed helped students better acquire the domain-independent help-seeking skills.
	~~Although students’ hint usage improved~~, ~~no major improvements in the deliberateness of students’ solution attempts~~ were ~~found (besides that indicated by the differences~~ in ~~hint-to-error ratio). It may be that~~ the Help ~~Tutor~~ did not ~~support~~ this ~~aspect of learning well enough~~. ~~An alternative explanation is that as long as students attempted to solve problems~~ (~~whether successfully or not~~) ~~they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources~~) to the ~~Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in~~ the ~~context of~~ domain ~~learning~~.

	[[Image:four months of HT.jpg]]		[[Image:four months of HT.jpg]]

Idoroll: /* Further Information */

2009-05-21T04:22:09Z

Further Information

← Older revision		Revision as of 04:22, 21 May 2009
Line 261:		Line 261:

	=== Further Information ===		=== Further Information ===
	~~Plans for June 2007 - Dec. 2007:~~
	* Present the study in the International Conference on Artificial Intelligence on Education
	* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
	* Analyze the logfiles for Metacognitive learning

	=== Annotated bibliography ===		=== Annotated bibliography ===

Idoroll: /* Explanation */

2009-05-21T04:21:49Z

Explanation

Show changes

Idoroll: /* Summary of results */

2009-05-21T04:20:24Z

Summary of results

← Older revision		Revision as of 04:20, 21 May 2009
Line 230:		Line 230:

	Overall, the following pattern of results emerges from the studies:		Overall, the following pattern of results emerges from the studies:
	- The Help Seeking Support Environment intervened appropriate actions during the learning process		* The Help Seeking Support Environment intervened appropriate actions during the learning process.
	- Students improved their help-seeking behavior while working with the system		* Students improved their help-seeking behavior while working with the system.
	- ~~Students~~ acquired better help-seeking declarative knowledge ~~following the system~~		* Students demonstrated improved help-seeking behavior even once support was removed, after working with the Help Tutor along two months and two different topics.
	- However, students' domain learning did not improve		* Furthermore, students acquired better help-seeking declarative knowledge.
	~~- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system~~.		* However, students' domain learning did not improve.

	=== Explanation ===		=== Explanation ===

Idoroll: /* Evaluation of goal 4: Improve future metacognitive behavior */

2009-05-21T04:18:23Z

Evaluation of goal 4: Improve future metacognitive behavior

← Older revision		Revision as of 04:18, 21 May 2009
Line 197:		Line 197:
	==== Evaluation of goal 4: Improve future metacognitive behavior ====		==== Evaluation of goal 4: Improve future metacognitive behavior ====

	~~To evaluate whether~~ the ~~effect~~ of the help-seeking ~~curriculum persists beyond~~ the ~~tutored environment~~, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.		Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
			The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
			Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

			[[Image:four months of HT.jpg]]



			In addition, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

	Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.		Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

Idoroll: Undo revision 9497 by Idoroll (Talk)

2009-05-21T04:16:55Z

Undo revision 9497 by Idoroll (Talk)

← Older revision		Revision as of 04:16, 21 May 2009
Line 173:		Line 173:
	==== Evaluation of goal 2: Improve metacognitive behavior ====		==== Evaluation of goal 2: Improve metacognitive behavior ====

			Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
			The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
			Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

	[[Image:help-seeking_behavior.jpg]]		[[Image:help-seeking_behavior.jpg]]

Idoroll at 04:16, 21 May 2009

2009-05-21T04:16:23Z

← Older revision		Revision as of 04:16, 21 May 2009
Line 175:		Line 175:


	[[Image:~~four months of HT~~.jpg]]		[[Image:help-seeking_behavior.jpg]]

			This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

	==== Evaluation of goal 3: Improve domain learning ====		==== Evaluation of goal 3: Improve domain learning ====

Idoroll: /* Evaluation of goal 2: Improve metacognitive behavior */

2009-05-21T04:14:48Z

Evaluation of goal 2: Improve metacognitive behavior

← Older revision		Revision as of 04:14, 21 May 2009
Line 173:		Line 173:
	==== Evaluation of goal 2: Improve metacognitive behavior ====		==== Evaluation of goal 2: Improve metacognitive behavior ====

	Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
	The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
	Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

	[[Image:four months of HT.jpg]]		[[Image:four months of HT.jpg]]