In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input

From LearnLab
Revision as of 11:02, 7 October 2010 by Koedinger (talk | contribs) (Undo revision 11082 by Alexsong (Talk))
Jump to: navigation, search

Lisa Anthony, Jie Yang, Kenneth R. Koedinger

Summary Table

PIs Lisa Anthony, Jie Yang, & Ken Koedinger
Other Contributers n/a
Study Start Date April 11, 2007
Study End Date May 25, 2007
LearnLab Site Central Westmoreland Career & Technology Center (CWCTC) and Wilkinsburg High School
LearnLab Course Algebra
Number of Students est. 102
Total Participant Hours est. 300
DataShop To be completed when study ends

Abstract

This in vivo classroom experiment compared differences in learning that occur depending on the modality of input during algebra equation solving. The key to this study was that the interface used was the normal Cognitive Tutor Algebra equation solver that students normally use in their classroom.

The hypothesis of this study was that, in addition to previously seen usability advantages of handwriting over typing in terms of speed and user satisfaction, handwriting will also provide learning advantages. We hypothesize two interrelated factors would be responsible for these advantages: (1) the improved support of handwriting for 2D mathematics notations such as fractions and exponents which can be difficult to represent and manipulate via the keyboard; and (2) the decrease in extraneous and irrelevant cognitive load due to removing the overhead a cumbersome menu-based interface for mathematics can provide.

Results from our preliminary lab study indicate that students achieve similar learning gains but finish in about half the time when they use handwriting vs using typing.

Glossary

Forthcoming, but will probably include

  • Sample worked-out-example:

Lanthony-example-unit18.gif

  • Learning rate/efficiency

Research question

How is robust learning affected by the modality of the generated input of students, specifically comparing handwriting and typing?

Background & Significance

Prior work has found that handwriting can be faster and more liked by users than using a keyboard and mouse for entering mathematics on the computer [1]. Anecdotal evidence suggests that students take a long time to learn an interface, possibly because it interferes with learning the goal concept. If handwriting can be shown to provide robust learning gains over traditional interfaces for mathematics, it may be possible to improve intelligent tutoring systems for mathematics by incorporating handwriting interfaces; students will be faster, more engaged and more deeply involved in knowledge construction during the learning process.

Independent Variables

Three factors were varied:

  • Modality of input: free-form handwriting space vs keyboard-and-mouse solver interface
  • Type of feedback: step-targeted vs answer-targeted
  • Type of instruction: pure problem-solving vs problem-solving plus worked examples

The modality is the primary factor. However, due to limitations of handwriting recognition technology and the importance of providing correct feedback to students as they learn, we must also consider varying levels of feedback. Current Cognitive Tutor Algebra provides feedback at every step, but with handwriting input, we cannot have complete confidence that we interpreted the student's input correctly without more information. As a potential mitigating factor, we introduce worked examples to the tutor interface to provide a sort of feed-forward. We therefore have 4 conditions which explore this space and allow us to determine to which factor to attribute any differences between conditions.

Conditions
Modality Type of Feedback Type of Instruction
Condition 1 Typing Step-Targeted Pure Problem-Solving
Condition 2 Typing Step-Targeted Problem-Solving + Worked Examples
Condition 3 Typing Answer-Targeted Problem-Solving + Worked Examples
Condition 4 Handwriting Answer-Targeted Problem-Solving + Worked Examples

Hypothesis

The handwriting modality has been shown to be faster than typing for mathematics [1], and this corresponding speed-up in the classroom implies that more detailed study of current topics or further study of more advanced topics is possible than students otherwise would be able to achieve. In addition, students' cognitive overhead during writing should be less than typing, in which they must spend time to think about how to generate the desired input, whereas in handwriting this would come more naturally due to long practice. This decrease in cognitive overhead may result in increased normal learning and long-term retention.

Dependent variables

  • Normal post-test, near transfer, immediate: Students were given a 20-minute post-test after their sessions with the computer tutor had concluded.
  • Long-term retention, near transfer': 3 weeks after the students complete Unit 18 for the study, they will be given a 20-minute retention test consisting of problems isomorphic to those seen in the session.
  • Far transfer: Far transfer items such as 4-step problems were included on all tests.
  • Acceleration of future learning: We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 3-step problems, to determine if there were learning curve differences during training.

Mediating variable:

  • Cognitive load: We also used a scale modeled after Paas' [3] cognitive load self-report scale to ask students how much mental effort they spent during the study and whether they felt that this mental effort came more from the material or from the computer.

Findings

Findings are reported in Lisa Anthony's PhD thesis.

Explanation

This study is part of both the Refinement and Fluency and the Coordinative Learning clusters.

Refinement and Fluency

This study addresses two of the 9 core assumptions: (1) fluency from basics: for true fluency, higher level skills must be grounded on well-practiced lower level skills; and (2) immediacy of feedback: a corollary of the emphasis on in vivo evaluation, scheduling, and explicit instruction is the idea that immediate feedback, which is a strong point of computerized instruction, facilitates learning.

The fluency from basics element in this study is relevant to the idea that students and teachers use handwritten notations in math class extensively on paper tests and when working on the chalkboard. Learning a new interface is not the goal of a math classroom, but rather learning the concepts and operations is. Thus, extraneous cognitive load of students is increased while learning the interface and learning the math compete for resources.

The immediacy of feedback issue is represented in this study by the type of feedback used: step-targeted vs answer-targeted. Based on limitations of handwriting recognition technology, step-targeted feedback may require serious technical development effort to achieve. Answer-targeted feedback may not be as effective as step-targeted, but this study explores whether the potential drawback of this factor and the potential benefit of the examples factor (below) will balance out.

Coordinative Learning

This study belongs to the examples and explanations sub-group. This study focuses on presenting worked examples to students right alongside problem-solving, eventually fading them so that students solved problems on their own during tutor use as well.

Descendants

None.

Annotated Bibliography

Analysis and write-up in progress.

References

[1] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2005) "Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer." ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR, 4 Apr 2005, pp. 1184-1187.

[2] Anthony, Lisa; Yang, Jie; Koedinger, Kenneth R. (2007) "Benefits of Handwritten Input for Students Learning Algebra Equation Solving." To appear in Proceedings of International Conference on Artificial Intelligence in Education (AIEd 2007).

[3] Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429-434.

Further Information

Plans for June 2007-December 2007
  • Analyze data to determine effect of modality as mitigated by potential benefits of worked examples or potential drawbacks of answer-targeted feedback.
  • Write up results for publication in a learning science conference.
  • Based on results of this study, handwriting recognition enhancements will be performed and a summative evaluation of the prototype Handwriting Algebra Tutor will be conducted in vivo in 2007-2008.