The Help Tutor Roll Aleven McLaren

From LearnLab
Revision as of 03:55, 5 February 2007 by Idoroll (talk | contribs)
Jump to: navigation, search

Towards Tutoring Metacognition - The Case of Help Seeking

Ido Roll, Vincent Aleven, Bruce McLaren, Kenneth Koedinger

Abstract

While working with a tutoring system, students often use available help facilities in an unproductive fashion. When managing their learning process, students often ask for redundant help, or avoid required one. To improve students’ help-seeking behavior we built the Help Seeking Support Envrionment, which includes three components: - Direct help seeking instruction, given by the teacher - A Self-Assessment Tutor, to help students evaluate their own need for help – The Help Tutor - a domain-independent agent that can be added as an adjunct to Cognitive Tutors. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive help-seeking model and giving students appropriate feedback.

In a series of classroom evaluations the Help Tutor captured help-seeking errors that were associated with poorer learning and with poorer declarative and procedural knowledge components of help seeking. Also, students performed less help-seeking errors while working with the Help Tutor, and acquired better help seeking declarative knowledge. However, we did not find evidence that this led to an improvement in learning at the domain level or to better help-seeking behavior in a paper-and-pencil environment. We raise a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of help facilities and of metacognitive tutoring within ITS.

Background and Significance

Not only that teaching metacognition holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking KCs: The ability to identify the need for help, and to elicit appropriate assistance from the relevant resources. However, considerable evidence shows that metacognitive knowledge components are in need of better support. For example, while working with Intelligent Tutoring Systems (ITS), students try to "game the system" or do not self-explain enough. Also, research shows that students' help-seeking behavior is far from ideal. This study focuses on students' Help Seeking behavior. While students already have knowledge components which regulate their help seeking behavior, these are often faulty or shallow. For example, students often have the following shallow Help Seeking KC:

Faulty procedural knowledge components: Cognitive aspects:

 If I don’t know the answer => 
 I should guess

Motivational aspects:

 If I get the answer correct =>
 I achieved the goal

Social aspects:

 If I ask for help =>
 I am weak

Faulty declarative knowledge components:

 Asking for hints will always reduce my skill level
 Making an error is better than asking for a hint
 Only weak people ask for help

Rather than making the metacogntive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine the Help Seeking knowledge components and acquire better feature validly.

This study has three main contributions:

  1. To investigate the nature of help-seeking knowledge and its acquisition, by designing the help-seeking (meta)cognitive model and evaluating student's learning using it
  2. To investigate whether conventional means and established methods can be used to effectively teach metacognitive knowledge components.
  3. To develop a framework for defining goals for metacognitive tutoring and directly assess metacognitive learning.

Glossary

See Help Tutor Glossary

  • Help Avoidance: An action in which the student attempts to solve the problem, even though they are assumed to be needing some form of help. This can be determined by the students' skill estimation, recent history of errors on the same step, etc.
  • Help Abuse: An action in which the students asks for help even though she is knowledgeable enough to attempt the step on her own.
  • Bottom out hint: The Cognitive Tutors have contextualized hints which offer several levels of support for students. Bottom out hint refers to the last level of hint, which usually conveys the answer.
  • Metacognition: Metacognition is the set of skills which manages the learning process. The decision what activities to perform next is an example for a metacognitive question.
  • Gaming the system: Attempting to make progress within the curriculum without thinking through the material, for example, by repeatedly guessing or using bottom-out hints

Research question

Can conventional and well-established instructional principles in the domain level be used to tutor metacognitive knowledge components such as Help Seeking knowledge components?


Independent Variables

The study evaluates the following independent variables:

  • Direct and immediate feedback on Help Seeking errors
  • Explicit declarative help-seeking instruction
  • Preparatory self-assessment episodes, to help students identify their knowledge deficits.

Dependent variables

The study uses two levels of dependent measures:

  • Directly assessing Help Seeking skills
  • Indirectly assessing help-seeking skills through their contribution to domain learning

Assessments of learning geometry domain knowledge:

  • Normal: Problem solving and explanation items like those in the tutor's instruction.
  • Transfer: Data insufficiency (or "not enough information") items included in the post-test form.

Assessments of improved help-seeking skills:

  • Normal (within training): Analyzing log-files against a model of ideal help-seeking behavior
  • Transfer: Do students effectively use optional hints embedded within certain test items to improve their score (90% credit if right after using the hint)
  • Long-term retention - assessing help-seeking behavior on subsequent learning events.

Hypothesis

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will

and thus, in turn, will

  • Improve learning of domain knowledge by using those skills effectively.

Findings

From the first study:

  • Students help-seeking behavior, whether optimal or faulty, transfers well between environments.
  • Help Seeking feedback alone reduces help-seeking errors, but does not contribute to learning of HS skills, nor does it improve domain learning.

Analysis of data from the second study is in progress.

Explanation

Students do not have the skills, but we didn't teach them right. Students' behavior is actually good Students have them, but do not want to use them

While students demonstrated better help-seeking behavior, they did not learn more on the cognitive level. Several possible explanations (and remediations) for that are:

  • Lack of cognitive headroom
    • possible improvement: Preparatory self-assessment sessions
  • Feedback on errors alone is not sufficient for extracting the help-seeking principles
    • Explicit Help Seeking instruction
  • Deeply rooted behavior
    • Longer study, across several units
  • Faulty identification of productive learning paths
    • Improving model based on log file analysis
  • Ineffective hints
    • Shorter hint sequences


Connections

FAQ for teachers

The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the Rummel Scripted Collaborative Problem Solving and the Walker A Peer Tutoring Addition projects.

Annotated bibliography

  1. Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [pdf]
  2. Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [pdf]
  3. Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [pdf]
  4. Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [pdf]
  5. Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [pdf]
  6. Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [pdf]
  7. Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [pdf]
  8. Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [pdf]

Similar PSLC studies

The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring French Culture

The Help Tutor scaffolds the learning process in a similar way to a script. It instructs the student what actions should be performed at each point, depending on the students' and problems' characteristics. In that, it is somewhat similar to the study of Rummel et al. on Scripted Collaborative Problem Solving