Theory Wiki - User contributions [en]

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:55:20Z

Lisa-Anthony: /* Independent Variables */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example:
[[Image:lanthony-example-unit18.png]]
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of feedback: step-targeted vs answer-targeted
* Type of instruction: pure problem-solving vs problem-solving plus worked examples

The modality is the primary factor. However, due to limitations of handwriting recognition technology and the importance of providing correct feedback to students as they learn, we must also consider varying levels of feedback. Current Cognitive Tutor Algebra provides feedback at every step, but with handwriting input, we cannot have complete confidence that we interpreted the student's input correctly without more information. As a potential mitigating factor, we introduce worked examples to the tutor interface to provide a sort of feed-forward. We therefore have 4 conditions which explore this space and allow us to determine to which factor to attribute any differences between conditions.

===== Conditions =====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| || '''Modality''' || '''Type of Feedback''' || '''Type of Instruction'''
|-
| '''Condition 1''' || Typing || Step-Targeted || Pure Problem-Solving
|-
| '''Condition 2''' || Typing || Step-Targeted || Problem-Solving + Worked Examples
|-
| '''Condition 3''' || Typing || Answer-Targeted || Problem-Solving + Worked Examples
|-
| '''Condition 4''' || Handwriting || Answer-Targeted || Problem-Solving + Worked Examples
|}

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice. This decrease in cognitive overhead may result in increased normal learning and long-term retention.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

* ''Cognitive load'': We also used a scale modeled after Paas' [3] cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

[3] Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429-434.

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:50:12Z

Lisa-Anthony: /* Conditions */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example:
[[Image:lanthony-example-unit18.png]]
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

===== Conditions =====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| || '''Modality''' || '''Type of Feedback''' || '''Type of Instruction'''
|-
| '''Condition 1''' || Typing || Step-Targeted || Pure Problem-Solving
|-
| '''Condition 2''' || Typing || Step-Targeted || Problem-Solving + Worked Examples
|-
| '''Condition 3''' || Typing || Answer-Targeted || Problem-Solving + Worked Examples
|-
| '''Condition 4''' || Handwriting || Answer-Targeted || Problem-Solving + Worked Examples
|}

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice. This decrease in cognitive overhead may result in increased normal learning and long-term retention.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

* ''Cognitive load'': We also used a scale modeled after Paas' [3] cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

[3] Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429-434.

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:49:50Z

Lisa-Anthony: /* Independent Variables */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example:
[[Image:lanthony-example-unit18.png]]
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

===== Conditions =====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| || Modality || Type of Feedback || Type of Instruction
|-
| '''Condition 1''' || Typing || Step-Targeted || Pure Problem-Solving
|-
| '''Condition 2''' || Typing || Step-Targeted || Problem-Solving + Worked Examples
|-
| '''Condition 3''' || Typing || Answer-Targeted || Problem-Solving + Worked Examples
|-
| '''Condition 4''' || Handwriting || Answer-Targeted || Problem-Solving + Worked Examples
|}

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice. This decrease in cognitive overhead may result in increased normal learning and long-term retention.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

* ''Cognitive load'': We also used a scale modeled after Paas' [3] cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

[3] Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429-434.

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-04-23T03:44:42Z

Lisa-Anthony: /* Hypothesis */

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:41:59Z

Lisa-Anthony: /* Glossary */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example:
[[Image:lanthony-example-unit18.png]]
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice. This decrease in cognitive overhead may result in increased normal learning and long-term retention.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

* ''Cognitive load'': We also used a scale modeled after Paas' [3] cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

[3] Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429-434.

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

File:Lanthony-example-unit9.png

2007-04-23T03:39:24Z

Lisa-Anthony: Worked example as used in the Handwriting Algebra Tutor project in example study.

Worked example as used in the Handwriting Algebra Tutor project in example study.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T03:38:39Z

Lisa-Anthony: /* Glossary */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example:
[[Image:lanthony-example-lab.jpg]]
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===

Results from this study showed that students in the handwriting condition finished the curriculum in half the time of their typing counterparts (F(2,35)=11.05, p<0.0005). Yet there was no significant difference in their pre-to-post scores between conditions (F(2,35)=0.293, n.s.). Students appear to have learned just as much in about half the time! In a classroom situation, this would allow teachers to give students more practice or move on to more advanced material in the curriculum sooner.

There was also a significant interaction between modality and the appearance of fractions in a problem (F2,36=5.25, p<0.01), which implies that the advantages we’ve seen for handwriting only improve as the math gets more complex.

In their own words, students commented that handwriting “made it easier” and “takes a shorter time”—statements that lend support to the hypothesis that handwriting involves less extraneous cognitive load. While this is only a preliminary result, we plan to explore this further in later studies by including a structured self-report of student-perceived cognitive load, modeled after (Paas & Van Merrienboer, 1994), in which they asked students to rate their perceived amount of mental effort during various instructional paradigms.

In addition to these learning-related results, we found that speed differed by modality: students were two times faster in handwriting than in typing to complete the problem set given to them. Students also rated the handwriting condition more highly than the typing condition (70% chose it as their favorite modality), after having copied a set of given equations in both conditions.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses one of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

=== Descendants ===

None.

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]] which is currently ongoing.

File:Lanthony-example-lab.jpg

2007-04-23T03:38:20Z

Lisa-Anthony: Worked example as used in the Handwriting Example project lab study.

Worked example as used in the Handwriting Example project lab study.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T03:37:57Z

Lisa-Anthony: /* Glossary */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example:
[[Image:lanthony-example-lab.png]]
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===

Results from this study showed that students in the handwriting condition finished the curriculum in half the time of their typing counterparts (F(2,35)=11.05, p<0.0005). Yet there was no significant difference in their pre-to-post scores between conditions (F(2,35)=0.293, n.s.). Students appear to have learned just as much in about half the time! In a classroom situation, this would allow teachers to give students more practice or move on to more advanced material in the curriculum sooner.

There was also a significant interaction between modality and the appearance of fractions in a problem (F2,36=5.25, p<0.01), which implies that the advantages we’ve seen for handwriting only improve as the math gets more complex.

In their own words, students commented that handwriting “made it easier” and “takes a shorter time”—statements that lend support to the hypothesis that handwriting involves less extraneous cognitive load. While this is only a preliminary result, we plan to explore this further in later studies by including a structured self-report of student-perceived cognitive load, modeled after (Paas & Van Merrienboer, 1994), in which they asked students to rate their perceived amount of mental effort during various instructional paradigms.

In addition to these learning-related results, we found that speed differed by modality: students were two times faster in handwriting than in typing to complete the problem set given to them. Students also rated the handwriting condition more highly than the typing condition (70% chose it as their favorite modality), after having copied a set of given equations in both conditions.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses one of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

=== Descendants ===

None.

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]] which is currently ongoing.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-04-23T03:37:46Z

Lisa-Anthony: /* Glossary */

File:Lanthony-example.png

2007-04-23T03:35:25Z

Lisa-Anthony: Worked example from the Handwriting Algebra Tutor project.

Worked example from the Handwriting Algebra Tutor project.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T03:34:29Z

Lisa-Anthony: /* Glossary */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example:
[[Image:lanthony-example.png]]
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===

Results from this study showed that students in the handwriting condition finished the curriculum in half the time of their typing counterparts (F(2,35)=11.05, p<0.0005). Yet there was no significant difference in their pre-to-post scores between conditions (F(2,35)=0.293, n.s.). Students appear to have learned just as much in about half the time! In a classroom situation, this would allow teachers to give students more practice or move on to more advanced material in the curriculum sooner.

There was also a significant interaction between modality and the appearance of fractions in a problem (F2,36=5.25, p<0.01), which implies that the advantages we’ve seen for handwriting only improve as the math gets more complex.

In their own words, students commented that handwriting “made it easier” and “takes a shorter time”—statements that lend support to the hypothesis that handwriting involves less extraneous cognitive load. While this is only a preliminary result, we plan to explore this further in later studies by including a structured self-report of student-perceived cognitive load, modeled after (Paas & Van Merrienboer, 1994), in which they asked students to rate their perceived amount of mental effort during various instructional paradigms.

In addition to these learning-related results, we found that speed differed by modality: students were two times faster in handwriting than in typing to complete the problem set given to them. Students also rated the handwriting condition more highly than the typing condition (70% chose it as their favorite modality), after having copied a set of given equations in both conditions.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses one of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

=== Descendants ===

None.

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]] which is currently ongoing.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T03:33:48Z

Lisa-Anthony: /* Plans for June 2007-December 2007 */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===

Results from this study showed that students in the handwriting condition finished the curriculum in half the time of their typing counterparts (F(2,35)=11.05, p<0.0005). Yet there was no significant difference in their pre-to-post scores between conditions (F(2,35)=0.293, n.s.). Students appear to have learned just as much in about half the time! In a classroom situation, this would allow teachers to give students more practice or move on to more advanced material in the curriculum sooner.

There was also a significant interaction between modality and the appearance of fractions in a problem (F2,36=5.25, p<0.01), which implies that the advantages we’ve seen for handwriting only improve as the math gets more complex.

In their own words, students commented that handwriting “made it easier” and “takes a shorter time”—statements that lend support to the hypothesis that handwriting involves less extraneous cognitive load. While this is only a preliminary result, we plan to explore this further in later studies by including a structured self-report of student-perceived cognitive load, modeled after (Paas & Van Merrienboer, 1994), in which they asked students to rate their perceived amount of mental effort during various instructional paradigms.

In addition to these learning-related results, we found that speed differed by modality: students were two times faster in handwriting than in typing to complete the problem set given to them. Students also rated the handwriting condition more highly than the typing condition (70% chose it as their favorite modality), after having copied a set of given equations in both conditions.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses one of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

=== Descendants ===

None.

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]] which is currently ongoing.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T03:32:20Z

Lisa-Anthony: /* Findings */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===

Results from this study showed that students in the handwriting condition finished the curriculum in half the time of their typing counterparts (F(2,35)=11.05, p<0.0005). Yet there was no significant difference in their pre-to-post scores between conditions (F(2,35)=0.293, n.s.). Students appear to have learned just as much in about half the time! In a classroom situation, this would allow teachers to give students more practice or move on to more advanced material in the curriculum sooner.

There was also a significant interaction between modality and the appearance of fractions in a problem (F2,36=5.25, p<0.01), which implies that the advantages we’ve seen for handwriting only improve as the math gets more complex.

In their own words, students commented that handwriting “made it easier” and “takes a shorter time”—statements that lend support to the hypothesis that handwriting involves less extraneous cognitive load. While this is only a preliminary result, we plan to explore this further in later studies by including a structured self-report of student-perceived cognitive load, modeled after (Paas & Van Merrienboer, 1994), in which they asked students to rate their perceived amount of mental effort during various instructional paradigms.

In addition to these learning-related results, we found that speed differed by modality: students were two times faster in handwriting than in typing to complete the problem set given to them. Students also rated the handwriting condition more highly than the typing condition (70% chose it as their favorite modality), after having copied a set of given equations in both conditions.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses one of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

=== Descendants ===

None.

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[add link here]] which is currently ongoing.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:28:51Z

Lisa-Anthony: /* Hypothesis */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice. This decrease in cognitive overhead may result in increased normal learning and long-term retention.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

* ''Cognitive load'': We also used a scale modeled after Paas' [3] cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

[3] Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429-434.

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:27:14Z

Lisa-Anthony: /* Dependent variables */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

* ''Cognitive load'': We also used a scale modeled after Paas' [3] cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

[3] Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429-434.

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:26:54Z

Lisa-Anthony: /* References */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

* ''Cognitive load'': We also used a scale modeled after Paas and van Merrienboer's cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

[3] Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429-434.

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:25:22Z

Lisa-Anthony: /* Dependent variables */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

* ''Cognitive load'': We also used a scale modeled after Paas and van Merrienboer's cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T03:23:42Z

Lisa-Anthony: /* Explanation */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

===== Coordinative Learning =====

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T03:12:24Z

Lisa-Anthony: /* Explanation */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses one of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

=== Descendants ===

None.

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[add link here]] which is currently ongoing.

Handwriting Algebra Tutor

2007-04-23T02:44:33Z

Lisa-Anthony: /* Abstract */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===

Much work has been done in the field on modality of presentation: that is, how to best present instructional materials to students, both in the classroom and on the computer. This project’s focus is instead of modality of generation: how the modality in which students produce their solutions interacts with the cognitive processes involved in conceptualizing and working through a solution.

Most intelligent tutoring systems have relied on standard computer interface paradigms using the keyboard and mouse, even in mathematics. These interfaces are often idiosyncratic and quirky in their support for standard math notations such as two-dimensional constructs like fractions and exponents. The spatial relationships between characters has inherent and important meaning in mathematics, far more so than in other forms of writing, yet computer typing interfaces are ill-equipped to deal with them. Freeform handwriting input may be uniquely suited to entering mathematics input on the computer by virtue of its natural support for spatially variegated notations and its lack of constraints imposed on students producing a solution (allowing the problem-solving process to benefit from foundational fluency in handwriting).

Therefore, this project’s goal is to explore the ways in which the use of handwriting recognition in the interfaces of intelligent tutoring systems, already shown to be highly effective learning environments, can lead to improved learning gains. Our work has already shown that handwriting provides usability benefits in that speed of entry increases, user error decreases, and user satisfaction increases. We have also shown preliminary evidence that handwriting also provides learning benefits, in that students solving the same problems by handwriting as others who were typing took only half the time as their typing counterparts.

One concern with the use of handwriting in intelligent tutoring systems, however, is that recognition technology is not perfect. To the extent that we cannot be confident of correctly recognizing what the student is writing, we cannot provide detailed, step-targeted feedback. Therefore, a trade-off is clear between development effort to improve recognition accuracy and need to support step-targeted feedback. We attempt to address this trade-off by using worked examples, providing a sort of feed-forward instead and providing opportunities for students to use coordination of the given problem and the given example to improve their knowledge. To this end, this project consists of a number of studies that explore what the advantages of using handwriting are, what cognitive and learning factors contribute to these advantages, and how we can leverage these advantages in real tutoring systems with minimal technical development cost.

=== Descendents ===
'''Completed Experiments'''
*[[Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving]]
*[[Effect of adding simple worked examples to problem-solving in algebra learning]]

'''In-Progress Experiments'''
*[[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]]

'''Planned Experiments'''
*Summative evaluation of enhanced handwriting interface with step-targeted feedback

=== Annotated Bibliography ===

[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== References ===

=== Further Information ===

Handwriting Algebra Tutor

2007-04-23T02:43:19Z

Lisa-Anthony:

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===

Much work has been done in the field on modality of presentation: that is, how to best present instructional materials to students, both in the classroom and on the computer. This project’s focus is instead of modality of generation: how the modality in which students produce their solutions interacts with the cognitive processes involved in conceptualizing and working through a solution.

Most intelligent tutoring systems have relied on standard computer interface paradigms using the keyboard and mouse, even in mathematics. These interfaces are often idiosyncratic and quirky in their support for standard math notations such as two-dimensional constructs like fractions and exponents. The spatial relationships between characters has inherent and important meaning in mathematics, far more so than in other forms of writing, yet computer typing interfaces are ill-equipped to deal with them. Freeform handwriting input may be uniquely suited to entering mathematics input on the computer by virtue of its natural support for spatially variegated notations and its lack of constraints imposed on students producing a solution (allowing the problem-solving process to benefit from foundational fluency in handwriting).

Therefore, this project’s goal is to explore the ways in which the use of handwriting recognition in the interfaces of intelligent tutoring systems, already shown to be highly effective learning environments, can lead to improved learning gains. Our work has already shown that handwriting provides usability benefits in that speed of entry increases, user error decreases, and user satisfaction increases. We have also shown preliminary evidence that handwriting also provides learning benefits, in that students solving the same problems by handwriting as others who were typing took only half the time as their typing counterparts.

One concern with the use of handwriting in intelligent tutoring systems, however, is that recognition technology is not perfect. To the extent that we cannot be confident of correctly recognizing what the student is writing, we cannot provide detailed, step-targeted feedback. Therefore, a trade-off is clear between development effort to improve recognition accuracy and need to support step-targeted feedback. We attempt to address this trade-off by using worked examples, providing a sort of feed-forward instead and providing opportunities for students to use coordination of the given problem and the given example to improve their knowledge. To this end, this project consists of a number of studies that explore what the advantages of using handwriting are, what cognitive and learning factors contribute to these advantages, and how we can leverage these advantages in real tutoring systems with minimal technical development cost.

=== Descendents ===
'''Completed Experiments'''
*[[Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving]]
*[[Effect of adding simple worked examples to problem-solving in algebra learning]]

'''In-Progress Experiments'''
*[[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]]

'''Planned Experiments'''
*Summative evaluation of enhanced handwriting interface with step-targeted feedback

=== Annotated Bibliography ===

[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== References ===

=== Further Information ===

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T02:36:21Z

Lisa-Anthony: /* Abstract */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

===== Coordinative Learning =====

nd addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

Handwriting Algebra Tutor

2007-04-23T02:30:35Z

Lisa-Anthony: /* Annotated Bibliography */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===

Much work has been done in the field on modality of presentation: that is, how to best present instructional materials to students, both in the classroom and on the computer. This project’s focus is instead of modality of generation: how the modality in which students produce their solutions interacts with the cognitive processes involved in conceptualizing and working through a solution. Most intelligent tutoring systems have relied on standard computer interface paradigms using the keyboard and mouse, even in mathematics. These interfaces are often idiosyncratic and quirky in their support for standard math notations such as two-dimensional constructs like fractions and exponents. The spatial relationships between characters has inherent and important meaning in mathematics, far more so than in other forms of writing, yet computer typing interfaces are ill-equipped to deal with them. Freeform handwriting input may be uniquely suited to entering mathematics input on the computer by virtue of its natural support for spatially variegated notations and its lack of constraints imposed on students producing a solution (allowing the problem-solving process to benefit from foundational fluency in handwriting).
Therefore, this project’s goal is to explore the ways in which the use of handwriting recognition in the interfaces of intelligent tutoring systems, already shown to be highly effective learning environments, can lead to improved learning gains. Our work has already shown that handwriting provides usability benefits in that speed of entry increases, user error decreases, and user satisfaction increases. We have also shown preliminary evidence that handwriting also provides learning benefits, in that students solving the same problems by handwriting as others who were typing took only half the time as their typing counterparts.

One concern with the use of handwriting in intelligent tutoring systems, however, is that recognition technology is not perfect. To the extent that we cannot be confident of correctly recognizing what the student is writing, we cannot provide detailed, step-targeted feedback. Therefore, a trade-off is clear between development effort to improve recognition accuracy and need to support step-targeted feedback. We attempt to address this trade-off by using worked examples, providing a sort of feed-forward instead and providing opportunities for students to use coordination of the given problem and the given example to improve their knowledge. To this end, this project consists of a number of studies that explore what the advantages of using handwriting are, what cognitive and learning factors contribute to these advantages, and how we can leverage these advantages in real tutoring systems with minimal technical development cost.

=== Glossary ===

=== Research Question ===

=== Background/Significance ===

=== Dependent Variables ===

=== Independent Variables ===

=== Hypotheses ===

=== Findings ===
This project has overseen three studies since it began in November 2004. The first study, the Math Input Study, was a motivating study designed to explore what advantages, if any, handwriting-based input has for mathematics entry on the computer. Learning in a classroom setting is characterized by constraints on time, varying student motivation and engagement, and other factors not directly related to test scores. Our study showed that students who entered math equations via handwriting input were three times faster, were less prone to errors in input, and enjoyed their experience more. In the classroom, this can translate to increased depth or breadth of coverage by virtue of the extra time afforded, and to improve student motivation by virtue of their increased engagement. See (Anthony et al, 2005) for more details.

The second study, the Preliminary Learning Study, took the first study’s results one step further and applied handwriting-based input to a learning situation. In this study we compared students solving problems in a simple type-in interface with a handwriting input space; instruction was in the form of worked examples interspersed with problem solving, and feedback was answer-only (“Correct”/“Incorrect”). This was a laboratory study to determine whether or not novice math students engaged in a learning task would experience the same positive effects of using a handwriting interface over a keyboard one. In fact, students were faster: two times faster in handwriting than in typing to complete the problem set given to them. Students also rated the handwriting condition more highly than the typing condition (70% chose it as their favorite modality), after having copied a set of given equations in both conditions.

Results from this study showed that students in the handwriting condition finished the curriculum in half the time of their typing counterparts (Figure 1a, F2,35=11.05, p<0.0005). Yet there was no significant difference in their pre-to-post scores between conditions (Figure 1b, F2,35=0.293, n.s.). Students appear to have learned just as much in about half the time! In a classroom situation, this would allow teachers to give students more practice or move on to more advanced material in the curriculum sooner. There was also a significant interaction between modality and the appearance of fractions in a problem (F2,36=5.25, p<0.01), which implies that the advantages we’ve seen for handwriting only improve as the math gets more complex. In their own words, students commented that handwriting “made it easier” and “takes a shorter time”—statements that lend support to the hypothesis that handwriting involves less extraneous cognitive load. While this is only a preliminary result, we plan to explore this further in later studies by including a structured self-report of student-perceived cognitive load, modeled after (Paas & Van Merrienboer, 1994), in which they asked students to rate their perceived amount of mental effort during various instructional paradigms.

The third study, the Worked Examples Study, is currently ongoing. It is an in vivo study taking place at a local LearnLab school, CWCTC. Intelligent tutoring systems such as Cognitive Tutors have long incorporated directed step-by-step feedback throughout the problem-solving process; while this is considered to be a strength of the method, it has not been shown to be critical to effective learning. If such detailed feedback is not necessary for student success, the instructional paradigm can be altered when using handwriting input to prevent recognition errors from

Figure 1a. Time-on-task differences between the handwriting and typing conditions in the Preliminary Learning Study. Handwriting was two times faster than typing. Figure 1b. Learning as measured by gain from pre-test to post-test in the Preliminary Learning Study. No significant difference was seen between modalities.

interrupting the student. For example, we could use worked examples as a method of feed-forward to help students. The study underway is designed to begin to address this concern by comparing existing Cognitive Tutors that provide detailed feedback to the same systems that also provide worked examples during problem solving. We intend to analyze student learning as well as student hint and help-seeking during use of the tutoring system to determine whether students that are provided with worked examples use the hint facilities of the tutor less frequently.

These studies continue to inform our development of a theory of robust learning in that we have seen the first positive evidence in favor of handwriting interfaces for learning applications. Students using handwriting in the Preliminary Learning Study may have experienced a heightened sense of fluency with the mathematics they were learning, in that the interface was better able to allow them to more directly represent and manipulate the equations. In their own words, students praised handwriting for similar reasons. However, this needs to be isolated in greater detail in future studies. Eventually, we expect that students using our system will be able to both achieve greater fluency and engage in robust coordinative learning, that is, integrating the feedforward worked examples instruction along with the step-by-step directed feedback in order to construct a deeper understanding of the target concepts.

=== Descendents ===
'''Completed Experiments'''
*[[Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving]]
*[[Effect of adding simple worked examples to problem-solving in algebra learning]]

'''In-Progress Experiments'''
*[[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]]

'''Planned Experiments'''
*Summative evaluation of enhanced handwriting interface with step-targeted feedback

=== Annotated Bibliography ===

[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== References ===

=== Further Information ===

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T02:29:36Z

Lisa-Anthony: /* Annotated Bibliography */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

===== Coordinative Learning =====

nd addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

Handwriting Algebra Tutor

2007-04-23T02:28:51Z

Lisa-Anthony:

Effect of adding simple worked examples to problem-solving in algebra learning

2007-04-23T02:12:12Z

Lisa-Anthony: /* Further Information */

Refinement and Fluency

2007-04-23T02:11:01Z

Lisa-Anthony: /* Descendents */

== The PSLC Refinement and Fluency cluster ==

=== Abstract ===
The studies in this cluster concern the design and organization of instructional activities to facilitate the acquisition, [[refinement]], and fluent control of critical [[knowledge components]]. The research of the cluster addresses a series of core propositions, including but not limited to the following.

1. cognitive task analysis or knowledge component analysis: To design effective instruction, we must analyze learning tasks into their simplest components.

2. fluency from basics: For true fluency, higher level skills must be grounded on well-practiced lower level skills.

3. scheduling of practice: [[Optimized scheduling]] of [[practice]] uses principles of memory to maximize robust learning and achieve mastery.

4. [[explicit instruction]]: Explicit rule-based instruction facilitates the acquisition of specific skills, but only if the rules are simple.

5. [[implicit instruction]]: On the other hand, implicit instruction or exposure serves to foster the development of initial familiarity with larger patterns.

6. immediacy of feedback: A corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback facilitates learning.

7. [[cue validity]]: In both explicit and implicit instruction, cue validity plays a central role in determining ease of learning of knowledge components.

8. [[focusing]]: Instruction that focuses the learner's attention on valid cues leads to more robust learning than unfocused instruction or instruction that focuses on less valid cues.

9. learning to learn: The acquisition of skills such as analysis, help-seeking, or advance organizers can promote future learning.

10. [[transfer]]: A learner's earlier knowledge places strong constraints on new learning, promoting some forms of learning, while blocking others.

The overall hypothesis is that instruction that systematically reflects the complex [[features]] of targeted knowledge in relation to the learner’s existing knowledge leads to more robust learning than instruction that does not. The principle is that the gap between targeted knowledge and existing knowledge needs to be directly reflected in the organization of instructional events. This organization includes the structure of knowledge components selected for instruction, the scheduling of learning events, practice, recall opportunities, explicit and implicit presentations, and other activities.

This hypothesis can be rephrased in terms of the PSLC general hypothesis, which is that [[robust learning]] occurs when the [[learning event space]] is designed to include appropriate target paths, and when students are encouraged to take those paths. The studies in this cluster focus on the formulation of well specified target paths with highly predictable learning outcomes.

===Significance===
A core theme in this cluster is that instruction in basic skills can facilitate the acquisition and refinement of knowledge and prepare the learner for [[fluency]]-enhancing practice. Instruction that provides practice and feedback for basic skills on a schedule that closely matches observed student abilities is important for this goal, and can be effectively delivered by computer. In the area of second language learning, the strengths of computerized instruction are matched by certain weaknesses. In particular, computerized tutors are not yet good at speech recognition, making it difficult to assess student production. Moreover, contact with a human teacher can increase the breadth of language usage, as well as motivation. Therefore, an optimal environment for language learning would combine the strengths of computerized instruction with those of classroom instruction. It is possible that a similar analysis will apply to science and math.

=== Glossary ===
[[:Category:Refinement and Fluency|Refinement and Fluency]] glossary.

=== Research question ===
The overall research question is how can instruction optimally organize the presentation of complex targeted knowledge, taking into account the learner’s existing knowledge as well as an analysis of the target domain? In examining this general question, the studies focus on the following dimensions of instructional organization, among others: the demands placed on learners of specific knowledge components, the scheduling of practice, the timing and extent of explicit teaching events relative to implicit learning opportunities, and the role of feedback.

=== Independent variables ===
At a general level, the research varies the organization of instructional events. This organization variable is typically based on alternative analyses of task demands, relevant knowledge components, and learner background.

=== Dependent variables ===
The dependent variables in these studies assess learner performance during learning events and following learning. Typical measures are percentage correct and number of learning trials or time to reach a given standard of performance. Response times are also measured in some cases.

=== Hypotheses ===
The overall hypothesis is that instruction that systematically reflects the complex features of targeted knowledge in relation to the learner’s existing knowledge leads to more robust learning than instruction that does not. A corollary of this hypothesis is that learning is increased by instructional activities that require the learner to attend to the relevant knowledge components of a learning task.

Specific hypotheses about the organization of instruction derive from task analyses of specific domain knowledge and the existing knowledge of the learner. A background assumption for most studies is that fluency is grounded in well-practiced lower level skills. A few examples of specific hypotheses are as follows:

1. scheduling of practice hypothesis: The optimal scheduling of practice uses principles of memory consolidation to maximize robust learning and achieve mastery.

2. Resonance hypothesis: The acquisition of knowledge components can be facilitated by evoking associations between divergent coding systems.

3. [[explicit instruction]] hypothesis: Explicit rule-based instruction facilitates the acquisition of specific skills, but only if the rules are simple.

4. [[implicit instruction]] hypothesis: Implicit instruction or exposure serves to foster the development of initial familiarity with larger patterns.

5. Feedback hypothesis: Instruction that provides immediate, diagnostic feedback will be superior to instruction that does not.

6. cue validity hypothesis: In both explicit and implicit instruction, cue validity plays a central role in determining ease of learning of knowledge components.

7. [[Focusing]] hypothesis: Instruction that focuses the learner's attention on valid cues will lead to more robust learning than unfocused instruction or instruction that focuses on less valid cues.

8. learning to learn hypothesis: The acquisition of skills such as analysis, help-seeking, or advance organizers can promote future learning.

9. Learner knowledge hypothesis: A learner's existing knowledge places strong constraints on new learning, promoting some forms of learning, while blocking others.

=== Explanation ===
All knowledge involves content and procedures that are specific to a domain. An analysis of the domain reveals the complexities that a learner of a given background will face and the knowledge components that are part of the overall complexity. Accordingly, the organization of instruction is critical in allowing the learner to attend to the critical valid features of knowledge components and to integrated them in authentic performance. Acquiring valid features and strengthening their associations facilitates retrieval during subsequent assessment and instruction, leading to more robust learning. Additionally, robust learning is increased by the scheduling of learning events that promotes the [[long-term retention]] of the associations.

=== Descendents ===

Explicit instruction and manipulations of attention & discrimination
* [[Intelligent_Writing_Tutor | First language effects on second language grammar acquisition]] (Mitamura-Wylie)
* [[Learning the role of radicals in reading Chinese]] (Liu et al.)
* [[Basic skills training|French dictation training]] (MacWhinney)
* [[Chinese pinyin dictation]] (Zhang-MacWhinney)
*[[Handwriting Algebra Tutor]] (Anthony, Yang & Koedinger)
**[[Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving]] (completed) [Also in Coordinative Learning]
**[[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]] (in progress) [Also in Coordinative Learning]
*[[Note-Taking_Technologies | Note-taking Project Page (Bauer & Koedinger)]] [Also in Coordinative Learning]
**[[Note-Taking: Restriction and Selection]] (completed)
**[[Note-Taking: Focusing On Concepts]] (planned)
**[[Note-Taking: Focusing On Quantity]] (planned)
*[[A word-experience model of Chinese character learning]] (Reichle, Perfetti, & Liu)
*[[Learning a tonal language: Chinese]] (Wang, Perfetti, Liu)

Optimal scheduling & fluency pressure
* [[Optimizing the practice schedule]] (Pavlik et al.)
* [[French gender cues | French Grammatical Gender Cue Learning Through Optimized Practice]] (Presson-MacWhinney)
* [[Japanese fluency]] (Yoshimura-MacWhinney)
* [[Providing optimal support for robust learning of syntactic constructions in ESL]] (Levin, Frishkoff, De Jong, Pavlik)
* [[Fostering fluency in second language learning]] (De Jong, Perfetti)
* [[Using learning curves to optimize problem assignment]] (Cen & Koedinger)

Knowledge accessibility including background knowledge & knowledge suppression
* [[Using syntactic priming to increase robust learning]] (De Jong, Perfetti, DeKeyser)

Active processing including learner control
* [[Mental rotations during vocabulary training]] (Tokowicz-Degani)

Novel knowledge component and cognitive task analysis
* [[The_Help_Tutor__Roll_Aleven_McLaren|Tutoring a meta-cognitive skill: Help-seeking (Roll, Aleven & McLaren)]] [Also in Interactive Communication]
* [[Composition_Effect__Kao_Roll|What is difficult about composite problems? (Kao, Roll)]]
* [[Arithmetical fluency project]] (Fiez)

=== Annotated bibliography ===
Forthcoming

[[Category:Cluster]]

A Multimodal (Handwriting) Interface for Solving Equations

2007-04-23T02:08:35Z

Lisa-Anthony: A Multimodal (Handwriting) Interface for Solving Equations moved to Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving: disambiguating this study from others in the same project line

#REDIRECT [[Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving]]

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T02:08:35Z

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===

None.

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[add link here]] which is currently ongoing.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T02:07:00Z

Lisa-Anthony: /* Coordinative Learning */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

===== Coordinative Learning =====

nd addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===

None.

=== Annotated Bibliography ===

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

2007-04-23T02:06:21Z

Lisa-Anthony:

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || April 11, 2007
|-
| '''Study End Date''' || May 25, 2007
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || est. 102
|-
| '''Total Participant Hours''' || est. 300
|-
| '''DataShop''' || To be completed when study ends
|}

=== Abstract ===
This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
Three factors were varied:
* Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
* Type of instruction: pure problem-solving vs problem-solving plus worked examples
* Type of feedback: step-targeted vs answer-targeted

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.

* ''Far transfer'': Far transfer items such as 4-step problems were included on all tests.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of both the [[Refinement and Fluency]] and the [[Coordinative Learning]] clusters.

===== Refinement and Fluency =====

===== Coordinative Learning =====

and addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===

None.

=== Annotated Bibliography ===

=== References ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
* Write up results for publication in a learning science conference.
* Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.

Coordinative Learning

2007-04-23T01:44:17Z

Lisa-Anthony: /* Descendents */

== The PSLC Coordinative Learning cluster ==

=== Abstract ===
The studies in the Coordinative Learning cluster tend to focus on varying ''a)'' the types of information available to learning or ''b)'' the instructional methods that they employ. In particular, the studies focus on the impact of having learners coordinate two or more types. Given that the student has multiple [[sources]]/methods available, two factors that might impact learning are:

*What is the relationship between the content in the two sources or the content generated by the two methods? Our hypothesis is that the two sources or methods facilitate [[robust learning]] when a [[knowledge component]] is difficult to understand or absent in one and is present or easier to understand in the other.
*When and how does the student coordinate between the two sources or methods? Our hypothesis is that students should be encouraged to compare the two, perhaps by putting them close together in space or time.

At the micro-level, the overall hypothesis is that robust learning occurs when the [[learning event space]] has target paths whose [[sense making]] difficulties complement each other (as expressed in the first bullet above) and the students make path choices that take advantage of these [[complementary]] paths (as in the second bullet, above). This hypothesis is just a specialization of the [[Root_node|general PSLC hypothesis]] to this cluster.

=== Glossary ===
[[:Category:Coordinative Learning|Coordinative Learning]] glossary.

*'''[[Co-training]]'''
*'''[[Complementary]]'''
*'''[[Conceptual tasks]]'''
*'''[[Contiguity]]'''
*'''[[Coordination]]'''
*'''[[Ecological Control Group]]'''
*'''[[External representations]]'''
*'''[[Input sources ]]'''
*'''[[Instructional method]]'''
*'''[[Multimedia sources]]'''
*'''[[Procedural tasks]]'''
*'''[[Self-explanation]]'''
*'''[[Self-supervised learning]]'''
*'''[[Sources]]'''
*'''[[Strategies]]'''
*'''[[Unlabeled examples]]'''

=== Research question ===

When and how does coordinating multiple sources of information or lines of reasoning increase robust learning?

Two sub-groups of coordinative learning studies are exploring these more specific questions:

1) Visualizations and Multi-modal sources

When does adding visualizations or other multi-modal input enhance robust learning and how do we best support students in coordinating these sources?

2) Examples and Explanations

When and how should example study by combined and coordinated with problem solving to increase robust learning? When and how should explicit explanations be added or requested of students before, during, or after example study and problem solving practice?

=== Independent variables ===

*Content of the sources (e.g., pictures, diagrams, written text, audio, animation) or the encouraged lines of reasoning (e.g., example study, self-explanation, conceptual task, procedural task) and combinations

*Instructional activities designed to engage students in [[coordination]] (e.g., conceptual vs. [[procedural]] exercises, contiguous presentation of sources, [[self-explanation]])

=== Dependent variables ===
[[Normal post-test]] and measures of [[robust learning]].

=== Hypotheses ===
When students are given sources/methods whose [[sense making]] difficulties are complementary and they are engaged in coordinating the sources/methods, then their learning will be more robust than it would otherwise be.

=== Explanation ===

There are both [[sense making]] and [[foundational skill building]] explanations. From the sense making perspective, if the sources/methods yield complementary content and the student is engaged in coordinating them, then the student is more likely to successfully understand the instruction because if a student fails to understand one of the sources/methods, he can use the second to make sense of the first. From a foundational skill building perspective, attending to both sources/methods simultaneously associates [[features]] from both with the learned knowledge components, thus potentially increasing feature validity and hence robust learning.

=== Descendents ===

;Visualizations and Multi-modal sources
*[[Contiguous Representations for Robust Learning (Aleven & Butcher)]]
*[[Mapping Visual and Verbal Information: Integrated Hints in Geometry (Aleven & Butcher)]]
*[[Visual Representations in Science Learning | Visual Representations in Science Learning (Davenport, Klahr & Koedinger)]]
*[[Co-training of Chinese characters| Co-training of Chinese characters (Liu, Perfetti, Dunlap, Zi, Mitchell)]]
*[[Learning Chinese pronunciation from a “talking head”| Learning Chinese pronunciation from a “talking head” (Liu, Massaro, Dunlap, Wu, Chen,Chan, Perfetti)]] [Was in Refinement and Fluency]

;Examples and Explanations
*[[Booth | Knowledge component construction vs. recall (Booth, Siegler, Koedinger & Rittle-Johnson)]]
*[[Stoichiometry_Study | Studying the Learning Effect of Personalization and Worked Examples in the Solving of Stoichiometry Problems (McLaren, Koedinger & Yaron)]]
*[[Note-Taking_Technologies | Note-taking Project Page (Bauer & Koedinger)]]
**[[Note-Taking: Restriction and Selection]] (completed)
**[[Note-Taking: Coordination]] (planned)
*[[REAP_main | The REAP Project: Implicit and explicit instruction on word meanings (Juffs & Eskenazi)]]
*[[Help_Lite (Aleven, Roll)|Hints during tutored problem solving – the effect of fewer hint levels with greater conceptual content (Aleven & Roll)]]
*[[Handwriting Algebra Tutor]] (Anthony, Yang & Koedinger)
**[[Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving]] (completed)
**[[Effect of adding simple worked examples to problem-solving in algebra learning]] (completed, analysis in progress)
**[[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]] (in progress)
*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn)]]
*[[Does learning from worked-out examples improve tutored problem solving? | Does learning from worked-out examples improve tutored problem solving? (Renkl, Aleven & Salden)]] [Also in Interactive Communication]
*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn)]] [In Interactive Communication but also relevant here]

=== Annotated Bibliography ===
Much research in human and machine learning research has advocated various kinds of “multiples” to assist learning:
* multiple representations (e.g., machine learning: Liere & Tadepalli, 1997; human learning: Ainsworth & Van Labeke, in press),
* multiple strategies (e.g., machine learning: Michalski & Tecucci 1997; Saitta, Botta, & Neri, 1993; human learning: Klahr & Siegler, 1978);
* multiple learning tasks (e.g., machine learning: Caruana, 1997; Case, Jain, Ott, Sharma, & Stephan, 1998; human learning: Holland, Holyoak, Nisbett, & Thagard, 1986);
* multiple data sources (e.g., machine learning: Blum & Mitchell, 1998; Collins & Singer, 1999).

Experiments in human learning have demonstrated, for instance, that instruction that combines rules or principles and [[example]]s yields better results than either alone (Holland, Holyoak, Nisbett, & Thagard, 1986) or, for instance, iterative instruction of both [[Procedural tasks|procedures]] and [[Conceptual tasks|concepts]] better learning (Rittle-Johnson & Koedinger, 2002; Rittle-Johnson, Siegler, & Alibali, 2001).

Experiments in machine learning have demonstrated how more robust, generalizable learning can be achieved by training a single learner on ''multiple'' related tasks (Caruana 1997) or by training ''multiple'' learning systems on the same task (Blum & Mitchell 1998; Collins & Singer 1999; Muslea, Minton, & Knoblock, 2002). Blum and Mitchell (1998) provide both empirical results and a proof of the circumstances under which strategy combinations enhance learning. In particular, the [[co-training]] approach for combining multiple learning strategies yields better learning to the extent that the learning strategies produce “uncorrelated errors” – when one is wrong the other is often right. As an example of PSLC work, Donmez et al. (2005) demonstrate, using a multi-dimensional collaborative process analysis, that regularities across ''multiple'' codings of the same data can be exploited for the purpose of improving text classification accuracy for difficult codings.

An ambitious goal of PSLC is provide a rigorous causal theory of human learning results at the level of precision of machine learning research.

* Ainsworth, S.E. & Van Labeke (in press) Multiple Forms of Dynamic Representation. Learning and Instruction.
* Blum, A., & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In Proceedings of Eleventh Annual Conference on Computational Learning Theory (COLT), (pp. 92–100). New York: ACM Press. Available: citeseer.nj.nec.com/blum98combining.html
* Caruana, R. (1997). Multitask learning. Machine Learning 28(1), 41-75. Available: citeseer.nj.nec.com/caruana97multitask.html.
* Case, J., Jain, S., Ott, M., Sharma, A., & Stephan, F. (1998). Robust learning aided by context. In Proceedings of Eleventh Annual Conference on Computational Learning Theory (COLT), (pp. 44-55). New York: ACM Press.
* Collins, M., & Singer, Y. (1999). Unsupervised models for named entity classification. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (pp. 189–196).
* Donmez, P., Rose, C. P., Stegmann, K., Weinberger, A., and Fischer, F. (2005). Supporting CSCL with Automatic Corpus Analysis Technology, to appear in the Proceedings of Computer Supported Collaborative Learning.
* Holland, J. H., Holyoak, K. J., Nisbett, R. E., & Thagard, P. R. (1986). Induction: Processes of inference, learning, and discovery. Cambridge, MA: MIT Press.
* Klahr D., and Siegler R.S. (1978). The Representation of Children's Knowledge. In H.W. Reese and L.P. Lipsitt (Eds.), Advances in Child Development and Behavior, Academic Press, New York, NY, pp. 61-116.
* Liere, R., & Tadepalli, P. (1997). Active learning with committees for text categorization. In Proceedings of AAAI-97, 14th Conference of the American Association for Artificial Intelligence (pp. 591—596). Menlo Park, CA: AAAI Press.
* Michalski, R., & Tecuci, G. (Eds.) (1997). Machine learning: A multi-strategy approach. Morgan Kaufmann.
* Muslea, I., Minton, S., & Knoblock, C. (2002). Active + semi-supervised learning = robust multi-view learning. In Proceedings of ICML-2002. Sydney, Australia.
* Rittle-Johnson, B., Siegler, R. S., & Alibali, M. W. (2001). Developing conceptual understanding and procedural skill in mathematics: An iterative process. Journal of Educational Psychology, 93(2), 346–262.
* Rittle-Johnson, B., & Koedinger, K. R. (2002). Comparing instructional strategies for integrating conceptual and procedural knowledge. Paper presented at the Psychology of Mathematics Education, National, Athens, GA.
* Saitta, L., Botta, M., & Neri, F. (1993). Multi-strategy learning and theory revision. Machine Learning, 11(2/3), 153–172.
[[Category:Cluster]]

Effect of adding simple worked examples to problem-solving in algebra learning

2007-04-23T01:42:45Z

Lisa-Anthony: /* Further Information */

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T01:34:06Z

Lisa-Anthony: /* Descendants */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===

None.

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[add link here]] which is currently ongoing.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-04-23T01:33:49Z

Lisa-Anthony: /* Descendants */

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T01:33:22Z

Lisa-Anthony: /* Annotated Bibliography */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===
n/a

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[add link here]] which is currently ongoing.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T01:32:38Z

Lisa-Anthony: /* Annotated Bibliography */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===
n/a

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2004) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[add link here]] which is currently ongoing.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T01:30:38Z

Lisa-Anthony: /* Further Information */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===
n/a

=== Annotated Bibliography ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2004) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

=== Further Information ===
=====Plans for June 2007-December 2007=====

* This study has been completed, analyzed, submitted, and accepted for publication.
* The next steps in this line of research are relevant to [[add link here]] which is currently ongoing.

Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving

2007-04-23T01:24:10Z

Lisa-Anthony: Added summary table

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || Research Programmers/Associates: Thomas Bolster (Research Associate, CMU HCII)
|-
| '''Study Start Date''' || August 1, 2005
|-
| '''Study End Date''' || October 8, 2005
|-
| '''LearnLab Site''' || n/a
|-
| '''LearnLab Course''' || n/a
|-
| '''Number of Students''' || 48
|-
| '''Total Participant Hours''' || 1200
|-
| '''DataShop''' || No
|}

=== Abstract ===
This laboratory experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. Students copied and studied a worked-out algebra example line by line before then solving an analogous problem while referring to the example. One-third of the students entered their input into a plain text box (keyboard condition), another third entered their input into a blank writing space (handwriting condition), and the final third entered their input in the writing space while also speaking the steps out loud (handwriting-plus-speaking).

The hypothesis of this study was that, in addition to previously seen ''usability'' advantages of handwriting over typing in terms of speed and user satisfaction, handwriting would also provide ''learning'' advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Preliminary results indicate that the handwriting students finished in about half the time that the keyboard students took (14.7 minutes vs 27.0 minutes) and yet they performed just as well on the post-test. More detailed analyses are in progress on isolating the effects of modality on learning rate and/or learning efficiency.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example and/or screenshot of interaction in handwriting and typing
* Learning rate/efficiency

=== Research question ===
How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

=== Background & Significance ===
Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

=== Independent Variables ===
One independent variable was used:
* Modality of input: handwriting, typing, or handwriting-plus-speaking.

=== Hypothesis ===
The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': During training, examples alternated with problems, and the problems were solved in one of the 3 modalities/conditions. Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). Analyses of log data to determine error rate during training are in progress of being analyzed.

* ''Near transfer, retention'': After the session the students were given a 20-minute post-test consisting of problems isomorphic to those seen in the session. Handwriting students and typing students both achieved similar pre-post gains, but handwriting-plus-speaking students achieved much lower gains.

* ''Far transfer'': No far transfer items were included.

* ''Acceleration of future learning'': No acceleration of future learning measures were included in this laboratory study.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Refinement and Fluency|Refinement and Fluency]] cluster (was Coordinative Learning) and addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math conpete for resources.

The immediacy of feedback issue is not present in this study but rather in the overall project which doesn't have a node yet.

Not sure if this should be both CL and RF or just CL or just RF. We don't have two instructional activities or sources of information as CL requires...

=== Descendants ===
n/a

=== Further Information ===
[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2004) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-04-23T01:17:16Z

Lisa-Anthony: Added summary table

Coordinative Learning

2007-04-23T01:08:57Z

Lisa-Anthony: /* Descendents */

== The PSLC Coordinative Learning cluster ==

=== Abstract ===
The studies in the Coordinative Learning cluster tend to focus on varying ''a)'' the types of information available to learning or ''b)'' the instructional methods that they employ. In particular, the studies focus on the impact of having learners coordinate two or more types. Given that the student has multiple [[sources]]/methods available, two factors that might impact learning are:

*What is the relationship between the content in the two sources or the content generated by the two methods? Our hypothesis is that the two sources or methods facilitate [[robust learning]] when a [[knowledge component]] is difficult to understand or absent in one and is present or easier to understand in the other.
*When and how does the student coordinate between the two sources or methods? Our hypothesis is that students should be encouraged to compare the two, perhaps by putting them close together in space or time.

At the micro-level, the overall hypothesis is that robust learning occurs when the [[learning event space]] has target paths whose [[sense making]] difficulties complement each other (as expressed in the first bullet above) and the students make path choices that take advantage of these [[complementary]] paths (as in the second bullet, above). This hypothesis is just a specialization of the [[Root_node|general PSLC hypothesis]] to this cluster.

=== Glossary ===
[[:Category:Coordinative Learning|Coordinative Learning]] glossary.

*'''[[Co-training]]'''
*'''[[Complementary]]'''
*'''[[Conceptual tasks]]'''
*'''[[Contiguity]]'''
*'''[[Coordination]]'''
*'''[[Ecological Control Group]]'''
*'''[[External representations]]'''
*'''[[Input sources ]]'''
*'''[[Instructional method]]'''
*'''[[Multimedia sources]]'''
*'''[[Procedural tasks]]'''
*'''[[Self-explanation]]'''
*'''[[Self-supervised learning]]'''
*'''[[Sources]]'''
*'''[[Strategies]]'''
*'''[[Unlabeled examples]]'''

=== Research question ===

When and how does coordinating multiple sources of information or lines of reasoning increase robust learning?

Two sub-groups of coordinative learning studies are exploring these more specific questions:

1) Visualizations and Multi-modal sources

When does adding visualizations or other multi-modal input enhance robust learning and how do we best support students in coordinating these sources?

2) Examples and Explanations

When and how should example study by combined and coordinated with problem solving to increase robust learning? When and how should explicit explanations be added or requested of students before, during, or after example study and problem solving practice?

=== Independent variables ===

*Content of the sources (e.g., pictures, diagrams, written text, audio, animation) or the encouraged lines of reasoning (e.g., example study, self-explanation, conceptual task, procedural task) and combinations

*Instructional activities designed to engage students in [[coordination]] (e.g., conceptual vs. [[procedural]] exercises, contiguous presentation of sources, [[self-explanation]])

=== Dependent variables ===
[[Normal post-test]] and measures of [[robust learning]].

=== Hypotheses ===
When students are given sources/methods whose [[sense making]] difficulties are complementary and they are engaged in coordinating the sources/methods, then their learning will be more robust than it would otherwise be.

=== Explanation ===

There are both [[sense making]] and [[foundational skill building]] explanations. From the sense making perspective, if the sources/methods yield complementary content and the student is engaged in coordinating them, then the student is more likely to successfully understand the instruction because if a student fails to understand one of the sources/methods, he can use the second to make sense of the first. From a foundational skill building perspective, attending to both sources/methods simultaneously associates [[features]] from both with the learned knowledge components, thus potentially increasing feature validity and hence robust learning.

=== Descendents ===

;Visualizations and Multi-modal sources
*[[Contiguous Representations for Robust Learning (Aleven & Butcher)]]
*[[Mapping Visual and Verbal Information: Integrated Hints in Geometry (Aleven & Butcher)]]
*[[Visual Representations in Science Learning | Visual Representations in Science Learning (Davenport, Klahr & Koedinger)]]
*[[Co-training of Chinese characters| Co-training of Chinese characters (Liu, Perfetti, Dunlap, Zi, Mitchell)]]
*[[Learning Chinese pronunciation from a “talking head”| Learning Chinese pronunciation from a “talking head” (Liu, Massaro, Dunlap, Wu, Chen,Chan, Perfetti)]] [Was in Refinement and Fluency]

;Examples and Explanations
*[[Booth | Knowledge component construction vs. recall (Booth, Siegler, Koedinger & Rittle-Johnson)]]
*[[Stoichiometry_Study | Studying the Learning Effect of Personalization and Worked Examples in the Solving of Stoichiometry Problems (McLaren, Koedinger & Yaron)]]
*[[Note-Taking_Technologies | Note-taking Project Page (Bauer & Koedinger)]]
**[[Note-Taking: Restriction and Selection]] (completed)
**[[Note-Taking: Coordination]] (planned)
*[[REAP_main | The REAP Project: Implicit and explicit instruction on word meanings (Juffs & Eskenazi)]]
*[[Help_Lite (Aleven, Roll)|Hints during tutored problem solving – the effect of fewer hint levels with greater conceptual content (Aleven & Roll)]]
*[[Handwriting Algebra Tutor]] (Anthony, Yang & Koedinger)
**[[Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving]] (completed)
**[[Effect of adding simple worked examples to problem-solving in algebra learning]] (completed, analysis in progress)
**[[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input]] (planned)
*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn)]]
*[[Does learning from worked-out examples improve tutored problem solving? | Does learning from worked-out examples improve tutored problem solving? (Renkl, Aleven & Salden)]] [Also in Interactive Communication]
*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn)]] [In Interactive Communication but also relevant here]

=== Annotated Bibliography ===
Much research in human and machine learning research has advocated various kinds of “multiples” to assist learning:
* multiple representations (e.g., machine learning: Liere & Tadepalli, 1997; human learning: Ainsworth & Van Labeke, in press),
* multiple strategies (e.g., machine learning: Michalski & Tecucci 1997; Saitta, Botta, & Neri, 1993; human learning: Klahr & Siegler, 1978);
* multiple learning tasks (e.g., machine learning: Caruana, 1997; Case, Jain, Ott, Sharma, & Stephan, 1998; human learning: Holland, Holyoak, Nisbett, & Thagard, 1986);
* multiple data sources (e.g., machine learning: Blum & Mitchell, 1998; Collins & Singer, 1999).

Experiments in human learning have demonstrated, for instance, that instruction that combines rules or principles and [[example]]s yields better results than either alone (Holland, Holyoak, Nisbett, & Thagard, 1986) or, for instance, iterative instruction of both [[Procedural tasks|procedures]] and [[Conceptual tasks|concepts]] better learning (Rittle-Johnson & Koedinger, 2002; Rittle-Johnson, Siegler, & Alibali, 2001).

Experiments in machine learning have demonstrated how more robust, generalizable learning can be achieved by training a single learner on ''multiple'' related tasks (Caruana 1997) or by training ''multiple'' learning systems on the same task (Blum & Mitchell 1998; Collins & Singer 1999; Muslea, Minton, & Knoblock, 2002). Blum and Mitchell (1998) provide both empirical results and a proof of the circumstances under which strategy combinations enhance learning. In particular, the [[co-training]] approach for combining multiple learning strategies yields better learning to the extent that the learning strategies produce “uncorrelated errors” – when one is wrong the other is often right. As an example of PSLC work, Donmez et al. (2005) demonstrate, using a multi-dimensional collaborative process analysis, that regularities across ''multiple'' codings of the same data can be exploited for the purpose of improving text classification accuracy for difficult codings.

An ambitious goal of PSLC is provide a rigorous causal theory of human learning results at the level of precision of machine learning research.

* Ainsworth, S.E. & Van Labeke (in press) Multiple Forms of Dynamic Representation. Learning and Instruction.
* Blum, A., & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In Proceedings of Eleventh Annual Conference on Computational Learning Theory (COLT), (pp. 92–100). New York: ACM Press. Available: citeseer.nj.nec.com/blum98combining.html
* Caruana, R. (1997). Multitask learning. Machine Learning 28(1), 41-75. Available: citeseer.nj.nec.com/caruana97multitask.html.
* Case, J., Jain, S., Ott, M., Sharma, A., & Stephan, F. (1998). Robust learning aided by context. In Proceedings of Eleventh Annual Conference on Computational Learning Theory (COLT), (pp. 44-55). New York: ACM Press.
* Collins, M., & Singer, Y. (1999). Unsupervised models for named entity classification. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (pp. 189–196).
* Donmez, P., Rose, C. P., Stegmann, K., Weinberger, A., and Fischer, F. (2005). Supporting CSCL with Automatic Corpus Analysis Technology, to appear in the Proceedings of Computer Supported Collaborative Learning.
* Holland, J. H., Holyoak, K. J., Nisbett, R. E., & Thagard, P. R. (1986). Induction: Processes of inference, learning, and discovery. Cambridge, MA: MIT Press.
* Klahr D., and Siegler R.S. (1978). The Representation of Children's Knowledge. In H.W. Reese and L.P. Lipsitt (Eds.), Advances in Child Development and Behavior, Academic Press, New York, NY, pp. 61-116.
* Liere, R., & Tadepalli, P. (1997). Active learning with committees for text categorization. In Proceedings of AAAI-97, 14th Conference of the American Association for Artificial Intelligence (pp. 591—596). Menlo Park, CA: AAAI Press.
* Michalski, R., & Tecuci, G. (Eds.) (1997). Machine learning: A multi-strategy approach. Morgan Kaufmann.
* Muslea, I., Minton, S., & Knoblock, C. (2002). Active + semi-supervised learning = robust multi-view learning. In Proceedings of ICML-2002. Sydney, Australia.
* Rittle-Johnson, B., Siegler, R. S., & Alibali, M. W. (2001). Developing conceptual understanding and procedural skill in mathematics: An iterative process. Journal of Educational Psychology, 93(2), 346–262.
* Rittle-Johnson, B., & Koedinger, K. R. (2002). Comparing instructional strategies for integrating conceptual and procedural knowledge. Paper presented at the Psychology of Mathematics Education, National, Athens, GA.
* Saitta, L., Botta, M., & Neri, F. (1993). Multi-strategy learning and theory revision. Machine Learning, 11(2/3), 153–172.
[[Category:Cluster]]

Effect of adding simple worked examples to problem-solving in algebra learning

2007-02-26T18:27:42Z

Lisa-Anthony: /* Dependent variables */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===
This ''in vivo'' experiment compared differences in learning that occur when students problem solve vs when they problem solve aided by worked [[example]]s. Students worked in the standard Cognitive Tutor Algebra lesson on 2-step problems. Those in the worked examples condition copied the worked example given to them using the solver's interface the first time they saw a particular problem type (''i.e.'', ax+b=c or a/x=c); following that, an analogous example would appear each time the students solve a similar problem.

The hypothesis of this study was that students who were given the worked examples would experience improved learning in both normal learning and in terms of the [[robust learning]] measures of [[transfer]] and [[accelerated future learning]]. Copying the problem the first time the students encountered a particular problem type acts as additional scaffolding for students to solve the problems.

Results are forthcoming.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example

=== Research question ===
Is robust learning affected by the addition of scaffolded worked examples to the problem-solving process?

=== Background & Significance ===
...Worked examples studies undergone at PSLC and beyond...

See VanLehn's paper on students using examples -- copying vs. as feedback ...
Lefevre & Dicksen ... (1986). Cognition and Instruction.

See Koedinger & Aleven's Assistance Dilemma explanation ...

=== Independent Variables ===
One independent variable was used:
* Inclusion of worked example: present or not present.

=== Hypothesis ===
The inclusion of worked examples during the problem-solving process will have benefits for learning by virtue of the scaffolding provided by having the students copy the example the first time they see a particular problem type. ''more?''

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 15-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, [[retention]]'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there was any difference in performance at the start of that lesson.

* ''Far transfer'': Far transfer items such as 3-step problems and literal equations were included on the immediate post-test.

* ''[[Accelerated future learning]]'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Coordinative Learning]] cluster and addresses the examples and explanation sub-group.

The students were given examples throughout their use of the tutor. On the first instance of a particular problem type, students were asked to copy out a worked example; on subsequent instances, examples remained on the screen while students solved analogous problems.

=== Descendants ===

=== Further Information ===
Connected to [[A_Multimodal_%28Handwriting%29_Interface_for_Solving_Equations|A multimodal (handwriting) interface for solving equations]] in the [[Refinement and Fluency]] cluster.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-02-26T17:08:32Z

Lisa-Anthony: /* Abstract */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===
This ''in vivo'' experiment compared differences in learning that occur when students problem solve vs when they problem solve aided by [[unlabeled examples]]. Students worked in the standard Cognitive Tutor Algebra lesson on 2-step problems. Those in the worked examples condition copied the worked example given to them using the solver's interface the first time they saw a particular problem type (''i.e.'', ax+b=c or a/x=c); following that, an analogous example would appear each time the students solve a similar problem.

The hypothesis of this study was that students who were given the worked examples would experience improved learning in both normal learning and in terms of the [[robust learning]] measures of [[transfer]] and [[accelerated future learning]]. Copying the problem the first time the students encountered a particular problem type acts as additional scaffolding for students to solve the problems.

Results are forthcoming.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example

=== Research question ===
Is robust learning affected by the addition of scaffolded worked examples to the problem-solving process?

=== Background & Significance ===
...Worked examples studies undergone at PSLC and beyond...

=== Independent Variables ===
One independent variable was used:
* Inclusion of worked example: present or not present.

=== Hypothesis ===
The inclusion of worked examples during the problem-solving process will have benefits for learning by virtue of the scaffolding provided by having the students copy the example the first time they see a particular problem type. ''more?''

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 15-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, [[retention]]'': No long-term retention measures were included in this study.

* ''Far transfer'': Far transfer items such as 3-step problems and literal equations were included on the immediate post-test.

* ''[[Accelerated future learning]]'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Coordinative Learning]] cluster and addresses the examples and explanation sub-group.

The students were given examples throughout their use of the tutor. On the first instance of a particular problem type, students were asked to copy out a worked example; on subsequent instances, examples remained on the screen while students solved analogous problems.

=== Descendants ===

=== Further Information ===
Connected to [[A_Multimodal_%28Handwriting%29_Interface_for_Solving_Equations|A multimodal (handwriting) interface for solving equations]] in the [[Refinement and Fluency]] cluster.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-02-26T16:54:04Z

Lisa-Anthony: /* Dependent variables */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===
This ''in vivo'' experiment compared differences in learning that occur when students problem solve vs when they problem solve aided by [[unlabeled examples]]. Students worked in the standard Cognitive Tutor Algebra lesson on 2-step problems. Those in the worked examples condition copied the worked example given to them using the solver's interface the first time they saw a particular problem type (''i.e.'', ax+b=c or a/x=c).

The hypothesis of this study was that students who were given the worked examples would experience improved learning in both normal learning and in terms of the [[robust learning]] measures of [[transfer]] and [[accelerated future learning]]. Copying the problem the first time the students encountered a particular problem type acts as additional scaffolding for students to solve the problems.

Results are forthcoming.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example

=== Research question ===
Is robust learning affected by the addition of scaffolded worked examples to the problem-solving process?

=== Background & Significance ===
...Worked examples studies undergone at PSLC and beyond...

=== Independent Variables ===
One independent variable was used:
* Inclusion of worked example: present or not present.

=== Hypothesis ===
The inclusion of worked examples during the problem-solving process will have benefits for learning by virtue of the scaffolding provided by having the students copy the example the first time they see a particular problem type. ''more?''

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 15-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, [[retention]]'': No long-term retention measures were included in this study.

* ''Far transfer'': Far transfer items such as 3-step problems and literal equations were included on the immediate post-test.

* ''[[Accelerated future learning]]'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Coordinative Learning]] cluster and addresses the examples and explanation sub-group.

The students were given examples throughout their use of the tutor. On the first instance of a particular problem type, students were asked to copy out a worked example; on subsequent instances, examples remained on the screen while students solved analogous problems.

=== Descendants ===

=== Further Information ===
Connected to [[A_Multimodal_%28Handwriting%29_Interface_for_Solving_Equations|A multimodal (handwriting) interface for solving equations]] in the [[Refinement and Fluency]] cluster.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-02-26T16:52:43Z

Lisa-Anthony: /* Abstract */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===
This ''in vivo'' experiment compared differences in learning that occur when students problem solve vs when they problem solve aided by [[unlabeled examples]]. Students worked in the standard Cognitive Tutor Algebra lesson on 2-step problems. Those in the worked examples condition copied the worked example given to them using the solver's interface the first time they saw a particular problem type (''i.e.'', ax+b=c or a/x=c).

The hypothesis of this study was that students who were given the worked examples would experience improved learning in both normal learning and in terms of the [[robust learning]] measures of [[transfer]] and [[accelerated future learning]]. Copying the problem the first time the students encountered a particular problem type acts as additional scaffolding for students to solve the problems.

Results are forthcoming.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example

=== Research question ===
Is robust learning affected by the addition of scaffolded worked examples to the problem-solving process?

=== Background & Significance ===
...Worked examples studies undergone at PSLC and beyond...

=== Independent Variables ===
One independent variable was used:
* Inclusion of worked example: present or not present.

=== Hypothesis ===
The inclusion of worked examples during the problem-solving process will have benefits for learning by virtue of the scaffolding provided by having the students copy the example the first time they see a particular problem type. ''more?''

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 15-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': No long-term retention measures were included in this study.

* ''Far transfer'': Far transfer items such as 3-step problems and literal equations were included on the immediate post-test.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Coordinative Learning]] cluster and addresses the examples and explanation sub-group.

The students were given examples throughout their use of the tutor. On the first instance of a particular problem type, students were asked to copy out a worked example; on subsequent instances, examples remained on the screen while students solved analogous problems.

=== Descendants ===

=== Further Information ===
Connected to [[A_Multimodal_%28Handwriting%29_Interface_for_Solving_Equations|A multimodal (handwriting) interface for solving equations]] in the [[Refinement and Fluency]] cluster.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-02-26T16:52:24Z

Lisa-Anthony: /* Abstract */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===
This ''in vivo'' experiment compared differences in learning that occur when students problem solve vs when they problem solve aided by [[unlabeled examples]]. Students worked in the standard Cognitive Tutor Algebra lesson on 2-step problems. Those in the worked examples condition copied the worked example given to them using the solver's interface the first time they saw a particular problem type (''i.e.'', ax+b=c or a/x=c).

The hypothesis of this study was that students who were given the worked examples would experience improved learning in both ''normal'' learning and in terms of the [[''robust'' learning]] measures of [[transfer]] and [[accelerated future learning]]. Copying the problem the first time the students encountered a particular problem type acts as additional scaffolding for students to solve the problems.

Results are forthcoming.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example

=== Research question ===
Is robust learning affected by the addition of scaffolded worked examples to the problem-solving process?

=== Background & Significance ===
...Worked examples studies undergone at PSLC and beyond...

=== Independent Variables ===
One independent variable was used:
* Inclusion of worked example: present or not present.

=== Hypothesis ===
The inclusion of worked examples during the problem-solving process will have benefits for learning by virtue of the scaffolding provided by having the students copy the example the first time they see a particular problem type. ''more?''

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 15-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': No long-term retention measures were included in this study.

* ''Far transfer'': Far transfer items such as 3-step problems and literal equations were included on the immediate post-test.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Coordinative Learning]] cluster and addresses the examples and explanation sub-group.

The students were given examples throughout their use of the tutor. On the first instance of a particular problem type, students were asked to copy out a worked example; on subsequent instances, examples remained on the screen while students solved analogous problems.

=== Descendants ===

=== Further Information ===
Connected to [[A_Multimodal_%28Handwriting%29_Interface_for_Solving_Equations|A multimodal (handwriting) interface for solving equations]] in the [[Refinement and Fluency]] cluster.

Effect of adding simple worked examples to problem-solving in algebra learning

2007-02-26T16:49:21Z

Lisa-Anthony:

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Abstract ===
This in vivo experiment compared differences in learning that occur when students problem solve vs when they problem solve aided by simple (unannotated) worked examples. Students worked in the standard Cognitive Tutor Algebra lesson on 2-step problems. Those in the worked examples condition copied the worked example given to them using the solver's interface the first time they saw a particular problem type (i.e., ax+b=c or a/x=c).

The hypothesis of this study was that students who were given the worked examples would experience improved learning in both ''normal'' learning and in terms of the ''[[robust]]'' learning measures of [[transfer]] and [[accelerated future learning]]. Copying the problem the first time the students encountered a particular problem type acts as additional scaffolding for students to solve the problems.

Results are forthcoming.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example

=== Research question ===
Is robust learning affected by the addition of scaffolded worked examples to the problem-solving process?

=== Background & Significance ===
...Worked examples studies undergone at PSLC and beyond...

=== Independent Variables ===
One independent variable was used:
* Inclusion of worked example: present or not present.

=== Hypothesis ===
The inclusion of worked examples during the problem-solving process will have benefits for learning by virtue of the scaffolding provided by having the students copy the example the first time they see a particular problem type. ''more?''

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 15-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, retention'': No long-term retention measures were included in this study.

* ''Far transfer'': Far transfer items such as 3-step problems and literal equations were included on the immediate post-test.

* ''Acceleration of future learning'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Coordinative Learning]] cluster and addresses the examples and explanation sub-group.

The students were given examples throughout their use of the tutor. On the first instance of a particular problem type, students were asked to copy out a worked example; on subsequent instances, examples remained on the screen while students solved analogous problems.

=== Descendants ===

=== Further Information ===
Connected to [[A_Multimodal_%28Handwriting%29_Interface_for_Solving_Equations|A multimodal (handwriting) interface for solving equations]] in the [[Refinement and Fluency]] cluster.

Coordinative Learning

2007-02-26T16:32:55Z

Lisa-Anthony: /* Descendents */

Refinement and Fluency

2007-02-26T16:30:53Z

Lisa-Anthony: /* Descendents */

== The PSLC Refinement and Fluency cluster ==

=== Abstract ===
The studies in this cluster concern the design and organization of instructional activities to facilitate the acquisition, [[refinement]], and fluent control of critical [[knowledge components]]. The research of the cluster addresses a series of core propositions, including but not limited to the following.

1. task analysis: To design effective instruction, we must analyze learning tasks into their simplest components.

2. fluency from basics: For true fluency, higher level skills must be grounded on well-practiced lower level skills.

3. scheduling of practice: The optimal scheduling of practice uses principles of memory [[consolidation]] to maximize robust learning and achieve mastery.

4. [[explicit instruction]]: Explicit rule-based instruction facilitates the acquisition of specific skills, but only if the rules are simple.

5. [[implicit instruction]]: On the other hand, implicit instruction or exposure serves to foster the development of initial familiarity with larger patterns.

6. immediacy of feedback: A corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback facilitates learning.

7. [[cue validity]]: In both explicit and implicit instruction, cue validity plays a central role in determining ease of learning of knowledge components.

8. focusing: Instruction that focuses the learner's attention on valid cues leads to more robust learning than unfocused instruction or instruction that focuses on less valid cues.

9. learning to learn: The acquisition of skills such as analysis, help-seeking, or advance organizers can promote future learning.

10. [[transfer]]: A learner's earlier knowledge places strong constraints on new learning, promoting some forms of learning, while blocking others.

The overall hypothesis is that instruction that systematically reflects the complex [[features]] of targeted knowledge in relation to the learner’s existing knowledge leads to more robust learning than instruction that does not. The principle is that the gap between targeted knowledge and existing knowledge needs to be directly reflected in the organization of instructional events. This organization includes the structure of knowledge components selected for instruction, the scheduling of learning events, practice, recall opportunities, explicit and implicit presentations, and other activities.

This hypothesis can be rephrased in terms of the PSLC general hypothesis, which is that [[robust learning]] occurs when the [[learning event space]] is designed to include appropriate target paths, and when students are encouraged to take those paths. The studies in this cluster focus on the formulation of well specified target paths with highly predictable learning outcomes.

===Significance===
A core theme in this cluster is that instruction in basic skills can facilitate the acquisition and refinement of knowledge and prepare the learner for [[fluency]]-enhancing practice. Instruction that provides practice and feedback for basic skills on a schedule that closely matches observed student abilities is important for this goal, and can be effectively delivered by computer. In the area of second language learning, the strengths of computerized instruction are matched by certain weaknesses. In particular, computerized tutors are not yet good at speech recognition, making it difficult to assess student production. Moreover, contact with a human teacher can increase the breadth of language usage, as well as motivation. Therefore, an optimal environment for language learning would combine the strengths of computerized instruction with those of classroom instruction. It is possible that a similar analysis will apply to science and math.

=== Glossary ===
[[:Category:Refinement and Fluency|Refinement and Fluency]] glossary.

=== Research question ===
The overall research question is how can instruction optimally organize the presentation of complex targeted knowledge, taking into account the learner’s existing knowledge as well as an analysis of the target domain? In examining this general question, the studies focus on the following dimensions of instructional organization, among others: the demands placed on learners of specific knowledge components, the scheduling of practice, the timing and extent of explicit teaching events relative to implicit learning opportunities, and the role of feedback.

=== Independent variables ===
At a general level, the research varies the organization of instructional events. This organization variable is typically based on alternative analyses of task demands, relevant knowledge components, and learner background.

=== Dependent variables ===
The dependent variables in these studies assess learner performance during learning events and following learning. Typical measures are percentage correct and number of learning trials or time to reach a given standard of performance. Response times are also measured in some cases.

=== Hypotheses ===
The overall hypothesis is that instruction that systematically reflects the complex features of targeted knowledge in relation to the learner’s existing knowledge leads to more robust learning than instruction that does not. A corollary of this hypothesis is that learning is increased by instructional activities that require the learner to attend to the relevant knowledge components of a learning task.

Specific hypotheses about the organization of instruction derive from task analyses of specific domain knowledge and the existing knowledge of the learner. A background assumption for most studies is that fluency is grounded in well-practiced lower level skills. A few examples of specific hypotheses are as follows:

1. scheduling of practice hypothesis: The optimal scheduling of practice uses principles of memory consolidation to maximize robust learning and achieve mastery.

2. Resonance hypothesis: The acquisition of knowledge components can be facilitated by evoking associations between divergent coding systems.

3. explicit instruction hypothesis: Explicit rule-based instruction facilitates the acquisition of specific skills, but only if the rules are simple.

4. implicit instruction hypothesis: Implicit instruction or exposure serves to foster the development of initial familiarity with larger patterns.

5. Feedback hypothesis: Instruction that provides immediate, diagnostic feedback will be superior to instruction that does not.

6. cue validity hypothesis: In both explicit and implicit instruction, cue validity plays a central role in determining ease of learning of knowledge components.

7. Focusing hypothesis: Instruction that focuses the learner's attention on valid cues will lead to more robust learning than unfocused instruction or instruction that focuses on less valid cues.

8. learning to learn hypothesis: The acquisition of skills such as analysis, help-seeking, or advance organizers can promote future learning.

9. Learner knowledge hypothesis: A learner's existing knowledge places strong constraints on new learning, promoting some forms of learning, while blocking others.

=== Explanation ===
All knowledge involves content and procedures that are specific to a domain. An analysis of the domain reveals the complexities that a learner of a given background will face and the knowledge components that are part of the overall complexity. Accordingly, the organization of instruction is critical in allowing the learner to attend to the critical valid features of knowledge components and to integrated them in authentic performance. Acquiring valid features and strengthening their associations facilitates retrieval during subsequent assessment and instruction, leading to more robust learning. Additionally, robust learning is increased by the scheduling of learning events that promotes the [[long-term retention]] of the associations.

=== Descendents ===

* [[Using syntactic priming to increase robust learning]] (De Jong, Perfetti, DeKeyser)

* [[Learning the role of radicals in reading Chinese]] (Liu et al.)

* [[Basic skills training|French dictation training]] (MacWhinney)

*[[French gender cues]] (Presson-MacWhinney)

*[[Chinese pinyin dictation]] (Zhang-MacWhinney)

*[[Japanese fluency]] (Yoshimura-MacWhinney)

* [[Intelligent_Writing_Tutor | First language effects on second language grammar acquisition]] (Mitamura-Wylie)

* [[Optimizing the practice schedule]] (Pavlik-MacWhinney)

*[[The_Help_Tutor__Roll_Aleven_McLaren|Tutoring a meta-cognitive skill: Help-seeking (Roll, Aleven & McLaren)]] [Was in Coordinative Learning and in Interactive Communication]

*[[Composition_Effect__Kao_Roll|What is difficult about composite problems? (Kao, Roll)]]

* [[Mental rotations during vocabulary training]] (Tokowicz-Degani)

* [[arithmetical fluency project]] (Fiez)

* [[A_Multimodal_%28Handwriting%29_Interface_for_Solving_Equations| A multimodal (handwriting) interface for solving equations]] (Anthony, Yang, & Koedinger) [Was in CL]

* [[Providing optimal support for robust learning of syntactic constructions in ESL]] (Levin, Frishkoff, De Jong, Pavlik)

* [[Fostering fluency in second language learning]] (De Jong, Perfetti)

=== Annotated bibliography ===
Forthcoming

[[Category:Cluster]]