Geometry Greatest Hits - Revision history

Mbett: Reverted edits by Detraransdell (Talk); changed back to last version by Kirsten-Butcher

2011-09-07T14:31:07Z

Reverted edits by Detraransdell (Talk); changed back to last version by Kirsten-Butcher

New page

== Geometry Greatest Hits ==
=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Vincent Aleven, Ryan Baker, Kirsten Butcher, & Ron Salden
|-
| '''Other Contributers''' || Octav Popescu (Research Programmer, CMU HCII), Jessica Kalka (Research Associate, CMU HCII)

|-
| '''Study Start Date''' || January, 2009
|-
| '''Study End Date''' || March, 2009
|-
| '''LearnLab Site''' || Greenville, Riverview, Steel Valley
|-
| '''LearnLab Course''' || Geometry
|-
| '''Number of Students''' || 98
|-
| '''Total Participant Hours''' ||
|-
| '''DataShop''' || Log data soon to be uploaded and available in the DataShop
|}
=== Abstract ===
The main idea in the current project is to combine instructional interventions derived from four instructional principles. Each of these interventions has been shown to be effective in separate (PSLC) studies, and can be expected on theoretical grounds to be synergistic (or complementary). We hypothesize that instruction that simultaneously implements several principles will be dramatically more effective than instruction that does not implement any of the targeted principles (e.g. current common practice), especially if the principles are tied to different learning mechanisms. This project will test this hypothesis, focusing on the following four principles:

* [[Visual-verbal integration]] principle
* [[Worked example principle]]
* [[Prompted self-explanation principle]]
* [[Accurate knowledge estimates principle]]

Building on our prior work that tested these principles individually, we have created a new version of the Geometry Cognitive Tutor that implements these four principles. We have conducted an in-vivo experiment, and will conduct a lab experiment, to test the hypothesis that the combination of these principles produces a large effect size compared to the standard Cognitive Tutor, which does not support any of these principles, or supports them less strongly.

=== Background & Significance ===
The PSLC’s in-vivo methodology, as well as standard practice in learning science, generally focuses on testing one principle at a time. This approach is useful for understanding which principles work, how they work, and what their boundary conditions are. However, it is also useful to test combinations of principles, because it elucidates boundary conditions and explores the degree to which principles are complementary or synergistic.

Knowing which instructional interventions and principles are synergistic (as well as when interventions and principles do not have any additive effects) is also an important practical goal within the learning sciences. Instructional designers often use principles in combination (e.g. Anderson et al, 1995; Quintana et al, 2004); knowing which combinations are effective in concert is therefore pragmatically useful.

Intelligent Tutoring Systems have been proven to be more effective than typical classroom instruction.
Can principle-oriented research make them even more effective?
Can demonstrable impact in the classroom be strengthened by combining principles from successful in vivo studies?
And will such a combination lead to a large effect size?

=== Glossary ===

=== Hypotheses ===

;H1
: A tutor that uses multiple PSLC learning principles in combination, each of which have been validated to lead to better robust learning when applied, will achieve a significantly higher effect size compared to an unmodified tutor than the principles achieve on their own.

=== Completed experiments===

* In vivo study: A two-condition in-vivo study (comparing the baseline tutor to a modified tutor with all four improvements). Measures of learning gains (including robust learning measures) and learning efficiency (time taken to complete tutor) were utilized.

=== Planned experiments===
* Lab study (2 phases):
**(1) A two-condition study (comparing the baseline tutor to the modified tutor with all five improvements) testing overall student learning (including measures of robust learning) and efficiency in one tutor unit (Angles).
**(2) Think-aloud (lab) research to determine if worked-examples and visual interaction have the hypothesized, complementary process effects.

=== Independent variables ===

The Greatest Hits version of the tutor had the following features, which are supported by prior PSLC research

* integrated problem format (symbolic information integrated in the diagram; all interaction happens in the diagram)
[[Image:VisVerb.GIF]]
* non-interactive conceptual example sets at the beginning of each curricular unit
* interactive worked examples at the beginning of each curricular, faded in an individualized manner
[[Image:WorkedExample.GIF]]
* diagrammatic self-explanations of incorrect steps
* tuned knowledge-tracing parameters to achieve more better individualized problem sequences (avoiding over-practice and under-practice)
* employed new knowledge-tracing algorithm that estimated the probability of guesses and slips in a contextual manner (to improve the accuracy of student modeling, which in turn better individualized problem sequences)

The Greatest Hits version of the tutor is compared to a control condition, which features standard tutor interactions and instruction:
[[Image:ControlCondition.GIF]]

====Dependent Variables====
*Problem-Solving Items. 
Problem Solving items have a similar format to the tutor – students must use known information to calculate the measure of an angle and should justify their problem solving step with a relevant geometry rule. These problem solving items also contain several types of new tasks: First, student must make a solvability judgment to determine if enough information is known to solve the step. Second, for “false answers, students must explain how the unsolvable problem could be makes solvable. Third, for solvable items, students must explain which diagram elements apply to the geometry rule that was used in the problem solving step. 
[[Image:ProblemSolvingItem.GIF]]
 
*Reasoning Items. 
Students also complete reasoning items, which assess how well they understand the conceptual geometry relationships by which one feature is used to solve others. For these items, students should indicate whether they can find the angles of a certain geometry rule. 
[[Image:ReasoningItem.GIF]]

=== Results ===
Not yet available.

=== Explanation ===
=== Further Information ===
==== Connections ====

==== Annotated Bibliography ====
==== References ====

Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995).
Cognitive tutors: Lessons learned. The Journal of the Learning
Sciences, 4 (2) 167-207.

Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., Kyza, E., Edelson,
D. C., & Soloway, E. (2004). A scaffolding design framework for software to support
science inquiry. The Journal of the Learning Sciences, 13(3), 337-386.

==== Future Plans ====

Kirsten-Butcher: /* Independent variables */

2009-05-18T16:50:14Z

Independent variables

← Older revision		Revision as of 16:50, 18 May 2009
Line 64:		Line 64:

	* integrated problem format (symbolic information integrated in the diagram; all interaction happens in the diagram)		* integrated problem format (symbolic information integrated in the diagram; all interaction happens in the diagram)
			[[Image:VisVerb.GIF]]
	* non-interactive conceptual example sets at the beginning of each curricular unit		* non-interactive conceptual example sets at the beginning of each curricular unit
	* interactive worked examples at the beginning of each curricular, faded in an individualized manner		* interactive worked examples at the beginning of each curricular, faded in an individualized manner
			[[Image:WorkedExample.GIF]]
	* diagrammatic self-explanations of incorrect steps		* diagrammatic self-explanations of incorrect steps
	* tuned knowledge-tracing parameters to achieve more better individualized problem sequences (avoiding over-practice and under-practice)		* tuned knowledge-tracing parameters to achieve more better individualized problem sequences (avoiding over-practice and under-practice)
	* employed new knowledge-tracing algorithm that estimated the probability of guesses and slips in a contextual manner (to improve the accuracy of student modeling, which in turn better individualized problem sequences)		* employed new knowledge-tracing algorithm that estimated the probability of guesses and slips in a contextual manner (to improve the accuracy of student modeling, which in turn better individualized problem sequences)

			The Greatest Hits version of the tutor is compared to a control condition, which features standard tutor interactions and instruction:
			[[Image:ControlCondition.GIF]]

	====Dependent Variables====		====Dependent Variables====

Kirsten-Butcher: /* Dependent Variables */

2009-05-18T16:44:14Z

Dependent Variables

← Older revision		Revision as of 16:44, 18 May 2009
Line 71:		Line 71:

	====Dependent Variables====		====Dependent Variables====
	*<b>Problem-Solving ~~items~~.</b><br>		*<b>Problem-Solving Items.</b><br>
	Problem Solving items have a similar format to the tutor – students must use known information to calculate the measure of an angle and should justify their problem solving step with a relevant geometry rule. These problem solving items also contain several types of new tasks: First, student must make a solvability judgment to determine if enough information is known to solve the step. Second, for “false answers, students must explain how the unsolvable problem could be makes solvable. Third, for solvable items, students must explain which diagram elements apply to the geometry rule that was used in the problem solving step.<br>		Problem Solving items have a similar format to the tutor – students must use known information to calculate the measure of an angle and should justify their problem solving step with a relevant geometry rule. These problem solving items also contain several types of new tasks: First, student must make a solvability judgment to determine if enough information is known to solve the step. Second, for “false answers, students must explain how the unsolvable problem could be makes solvable. Third, for solvable items, students must explain which diagram elements apply to the geometry rule that was used in the problem solving step.<br>
	[[Image:ProblemSolvingItem.GIF]]		[[Image:ProblemSolvingItem.GIF]]
			<br><br>
			*<b>Reasoning Items.</b><br>
			Students also complete reasoning items, which assess how well they understand the conceptual geometry relationships by which one feature is used to solve others. For these items, students should indicate whether they can find the angles of a certain geometry rule.<br>
			[[Image:ReasoningItem.GIF]]

	=== Results ===		=== Results ===

Kirsten-Butcher: /* Independent variables */

2009-05-18T16:42:02Z

Independent variables

← Older revision		Revision as of 16:42, 18 May 2009
Line 69:		Line 69:
	* tuned knowledge-tracing parameters to achieve more better individualized problem sequences (avoiding over-practice and under-practice)		* tuned knowledge-tracing parameters to achieve more better individualized problem sequences (avoiding over-practice and under-practice)
	* employed new knowledge-tracing algorithm that estimated the probability of guesses and slips in a contextual manner (to improve the accuracy of student modeling, which in turn better individualized problem sequences)		* employed new knowledge-tracing algorithm that estimated the probability of guesses and slips in a contextual manner (to improve the accuracy of student modeling, which in turn better individualized problem sequences)

			====Dependent Variables====
			*<b>Problem-Solving items.</b><br>
			Problem Solving items have a similar format to the tutor – students must use known information to calculate the measure of an angle and should justify their problem solving step with a relevant geometry rule. These problem solving items also contain several types of new tasks: First, student must make a solvability judgment to determine if enough information is known to solve the step. Second, for “false answers, students must explain how the unsolvable problem could be makes solvable. Third, for solvable items, students must explain which diagram elements apply to the geometry rule that was used in the problem solving step.<br>
			[[Image:ProblemSolvingItem.GIF]]

	=== Results ===		=== Results ===

Ryan: /* Connections */

2009-05-16T15:11:57Z

Connections

← Older revision		Revision as of 15:11, 16 May 2009
Line 76:		Line 76:
	=== Further Information ===		=== Further Information ===
	==== Connections ====		==== Connections ====

	~~[[Baker Choices in LE Space]] -- project where work on estimating guess and slip was conducted.~~

	==== Annotated Bibliography ====		==== Annotated Bibliography ====

Ryan: /* Connections */

2009-05-16T15:11:04Z

Connections

← Older revision		Revision as of 15:11, 16 May 2009
Line 77:		Line 77:
	==== Connections ====		==== Connections ====

	[[Baker Choices in LE Space -- project where work on estimating guess and slip was conducted.		[[Baker Choices in LE Space]] -- project where work on estimating guess and slip was conducted.

	==== Annotated Bibliography ====		==== Annotated Bibliography ====

Ryan: /* Connections */

2009-05-16T15:10:53Z

Connections

← Older revision		Revision as of 15:10, 16 May 2009
Line 77:		Line 77:
	==== Connections ====		==== Connections ====

	[[Baker Choices in LE Space]]		[[Baker Choices in LE Space -- project where work on estimating guess and slip was conducted.

	==== Annotated Bibliography ====		==== Annotated Bibliography ====

Ryan: /* Connections */

2009-05-16T15:10:10Z

Connections

← Older revision		Revision as of 15:10, 16 May 2009
Line 76:		Line 76:
	=== Further Information ===		=== Further Information ===
	==== Connections ====		==== Connections ====

			[[Baker Choices in LE Space]]

	==== Annotated Bibliography ====		==== Annotated Bibliography ====
	==== References ====		==== References ====

Ryan at 02:38, 16 May 2009

2009-05-16T02:38:52Z

← Older revision		Revision as of 02:38, 16 May 2009
Line 23:		Line 23:
	\|}		\|}
	=== Abstract ===		=== Abstract ===
	The main idea in the current project is to combine instructional interventions derived from ~~five~~ instructional principles. Each of these interventions has been shown to be effective in separate (PSLC) studies, and can be expected on theoretical grounds to be synergistic (or complementary). We hypothesize that instruction that simultaneously implements several principles will be dramatically more effective than instruction that does not implement any of the targeted principles (e.g. current common practice), especially if the principles are tied to different learning mechanisms. This project will test this hypothesis, focusing on the following ~~five~~ principles:		The main idea in the current project is to combine instructional interventions derived from four instructional principles. Each of these interventions has been shown to be effective in separate (PSLC) studies, and can be expected on theoretical grounds to be synergistic (or complementary). We hypothesize that instruction that simultaneously implements several principles will be dramatically more effective than instruction that does not implement any of the targeted principles (e.g. current common practice), especially if the principles are tied to different learning mechanisms. This project will test this hypothesis, focusing on the following four principles:

	* [[Visual-verbal integration]] principle		* [[Visual-verbal integration]] principle
	* [[Worked example principle]]		* [[Worked example principle]]
	* [[Prompted self-explanation principle]]		* [[Prompted self-explanation principle]]
	* ~~Accurate knowledge decomposition principle (part of the complete and efficient practice principle)~~		* [[Accurate knowledge estimates principle]]
	* Accurate knowledge estimates principle ~~(part of the complete and efficient practice principle)~~

	Building on our prior work that tested these principles individually, we have created a new version of the Geometry Cognitive Tutor that implements these ~~five~~ principles. We have conducted an in-vivo experiment, and will conduct a lab experiment, to test the hypothesis that the combination of these principles produces a large effect size compared to the standard Cognitive Tutor, which does not support any of these principles, or supports them less strongly.		Building on our prior work that tested these principles individually, we have created a new version of the Geometry Cognitive Tutor that implements these four principles. We have conducted an in-vivo experiment, and will conduct a lab experiment, to test the hypothesis that the combination of these principles produces a large effect size compared to the standard Cognitive Tutor, which does not support any of these principles, or supports them less strongly.

	=== Background & Significance ===		=== Background & Significance ===
	PSLC’s in vivo methodology generally focuses on testing one principle at a time.		The PSLC’s in-vivo methodology, as well as standard practice in learning science, generally focuses on testing one principle at a time. This approach is useful for understanding which principles work, how they work, and what their boundary conditions are. However, it is also useful to test combinations of principles, because it elucidates boundary conditions and explores the degree to which principles are complementary or synergistic.
	This approach is useful for understanding which principles work, how they work, and what their boundary conditions are.
	~~Testing~~ combinations of principles ~~is useful also~~, because it elucidates boundary conditions and explores the degree to which principles are complementary or synergistic.

	Knowing which instructional interventions and principles are synergistic (as well as when interventions and principles do not have any additive effects) is also an important practical goal within the learning sciences. Instructional designers often use principles in combination (e.g. Anderson et al, 1995; Quintana et al, 2004); knowing which combinations are effective in concert is therefore pragmatically useful.		Knowing which instructional interventions and principles are synergistic (as well as when interventions and principles do not have any additive effects) is also an important practical goal within the learning sciences. Instructional designers often use principles in combination (e.g. Anderson et al, 1995; Quintana et al, 2004); knowing which combinations are effective in concert is therefore pragmatically useful.
Line 45:		Line 42:
	And will such a combination lead to a large effect size?		And will such a combination lead to a large effect size?

			=== Glossary ===

	~~=== Glossary ===~~
	=== Hypotheses ===		=== Hypotheses ===

			;H1
			: A tutor that uses multiple PSLC learning principles in combination, each of which have been validated to lead to better robust learning when applied, will achieve a significantly higher effect size compared to an unmodified tutor than the principles achieve on their own.

	=== Completed experiments===		=== Completed experiments===
	* In vivo study: A two-condition in-vivo study (comparing the baseline tutor to ~~the~~ modified tutor with all four improvements). Measures of learning gains and learning efficiency (time taken to complete tutor) ~~will be~~ utilized.
			* In vivo study: A two-condition in-vivo study (comparing the baseline tutor to a modified tutor with all four improvements). Measures of learning gains (including robust learning measures) and learning efficiency (time taken to complete tutor) were utilized.

	=== Planned experiments===		=== Planned experiments===
Line 77:		Line 78:
	==== Annotated Bibliography ====		==== Annotated Bibliography ====
	==== References ====		==== References ====

			Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995).
			Cognitive tutors: Lessons learned. The Journal of the Learning
			Sciences, 4 (2) 167-207.

			Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., Kyza, E., Edelson,
			D. C., & Soloway, E. (2004). A scaffolding design framework for software to support
			science inquiry. The Journal of the Learning Sciences, 13(3), 337-386.

	==== Future Plans ====		==== Future Plans ====

Aleven at 01:36, 15 May 2009

2009-05-15T01:36:29Z

← Older revision		Revision as of 01:36, 15 May 2009
Line 16:		Line 16:
	\| '''LearnLab Course''' \|\| Geometry		\| '''LearnLab Course''' \|\| Geometry
	\|-		\|-
	\| '''Number of Students''' \|\|		\| '''Number of Students''' \|\| 98
	\|-		\|-
	\| '''Total Participant Hours''' \|\|		\| '''Total Participant Hours''' \|\|