Difference between revisions of "The Help Tutor Roll Aleven McLaren"
(→Shallow help seeking knowledge components) |
(→Hypothesis) |
||
| Line 122: | Line 122: | ||
* Improve learning of domain knowledge by using those skills effectively. | * Improve learning of domain knowledge by using those skills effectively. | ||
| − | + | === Findings === | |
As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals: | As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals: | ||
Revision as of 04:14, 16 February 2007
Contents
Towards Tutoring Metacognition - The Case of Help Seeking
Ido Roll, Vincent Aleven, Bruce McLaren, Kenneth Koedinger
Abstract
While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate metacognitive process in doing so. For example, students often ask for help too frequently or not frequently enough. In this project we built an Intelligent Tutoring System to teach metacognition, and in particular, to improve students' help-seeking behavior. Our Help Seeking Support Environment includes three components:
- Direct help seeking instruction, given by the teacher
- A Self-Assessment Tutor, to help students evaluate their own need for help
- The Help Tutor - a domain-independent agent that can be added as an adjunct to a cognitive tutor. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive help-seeking model and giving students appropriate feedback.
In a series of in vivo experiments, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer declarative and procedural knowledge components of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking declarative knowledge components. However, we did not find evidence that this led to an improvement in learning at the domain level or to better help-seeking behavior in a paper-and-pencil environment. We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of help facilities and of metacognitive tutoring within Intelligent Tutoring Systems.
Background and Significance
Not only that teaching metacognition holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking knowledge components: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities]. However, considerable evidence shows that metacognitive knowledge components are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "game the system" or do not self-explain enough. Similarly, research shows that students' help-seeking behavior leaves much room for improvement.
Shallow help seeking knowledge components
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of help abuse, termed 'clicking through hints'), without taking enough time to read the requested hints. Extensive log-file analysis suggests that students apply faulty knowledge components such as the following:
Faulty procedural knowledge components: Cognitive aspects:
If I don’t know the answer => I should guess
Motivational aspects:
If I get the answer correct => I achieved the goal
Social aspects:
If I ask for help => I am weak
Faulty declarative knowledge components:
Asking for hints will always reduce my skill level
Making an error is better than asking for a hint
Only weak people ask for help
Teaching vs. supporting metacognition
Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.
Towards that end, we chose not to support students' help seeking actions, but to teach them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking knowledge components and acquire better feature validity of their help-seeking metacognitive skills.
By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.
Glossary
Research questions
- Can conventional and well-established instructional principles in the domain level be used to tutor metacognitive knowledge components such as Help Seeking knowledge components?
- Does the practice of better metacognitive behavior translates, in turn, to better domain learning?
In addition, the project makes the following contributions:
- An improved understanding of the nature of help-seeking knowledge and its acquisition.
- A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.
Independent Variables
Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support. Control condition used the conventional Geometry Cognitive Tutor
The treatment condition varied between studies:
- Study one: The Geometry Cognitive Tutor + the Help Tutor
- Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)
The Help Tutor:
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].
The Self-Assessment Tutor: The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills. The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.
Explicit help-seeking instruction: As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.
Dependent variables
The study uses two levels of dependent measures:
- Directly assessing Help Seeking skills
- Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.
1. Assessments of help-seeking konwledge:
- Normal post-test:
- Declarative: hypothetical help-seeking dilemmas
- Procedural: Help seeking error rate while working with the tutor
- Transfer: Ability to use optional hints embedded within certain test items in the paper test.
2. Assessments of domain konwledge:
- Normal post-test: Problem solving and explanation items like those in the tutor's instruction.
- Transfer:
- Data insufficiency (or "not enough information") items.
- Conceptual understanding items (study two only)
Hypothesis
The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
- Improve feature validity of students' help seeking skills
and thus, in turn, will
- Improve learning of domain knowledge by using those skills effectively.
Findings
As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
- First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
- Then, it should lead to an improved metacognitive behavior within the tutoring system.
- This, in turn, should lead to an improvement in the domain learning.
- The effect should persist beyond the scope of the tutoring system.
- As a result, students are expected to become better future learners.
Evaluation of goal 1: Capture metacognitive errors
In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test. This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.
File:Help-seeking and learning.png
Evaluation of goal 2: Improve metacognitive behavior
Log files from study 1 were analyzed to evaluate whether students improved their help-seeking behavior while working with the Help Tutor. While overall there was only a minor decrease in the error rate (from 19% in the control condition to 16% in the Help Tutor condition), there was a significant decrease in the error rate on first hints, following hints, and the portion of bottom-out hints out of all hint-sequences.
File:Help-seeking behavior.png
This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.
Evaluation of goal 3: Improve domain learning
While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.
Study 1 results:
Study 2 results:
Evaluation of goal 3: Improve domain learning
To evaluate the effect of the help-seeking curriculum, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.
Hypothetical hellp seeking dilemmas, such as the following
Explanation
Students do not have the skills, but we didn't teach them right. Students' behavior is actually good Students have them, but do not want to use them
While students demonstrated better help-seeking behavior, they did not learn more on the cognitive level. Several possible explanations (and remediations) for that are:
- Lack of cognitive headroom
- possible improvement: Preparatory self-assessment sessions
- Feedback on errors alone is not sufficient for extracting the help-seeking principles
- Explicit Help Seeking instruction
- Deeply rooted behavior
- Longer study, across several units
- Faulty identification of productive learning paths
- Improving model based on log file analysis
- Ineffective hints
- Shorter hint sequences
Connections
The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring French Culture
The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the Rummel Scripted Collaborative Problem Solving and the Walker A Peer Tutoring Addition projects.
Going to do an in-vivo study at a LearnLab site? Check out how to answer teacher's FAQ
Annotated bibliography
- Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [pdf]
- Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [pdf]
- Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [pdf]
- Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [pdf]
- Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [pdf]
- Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [pdf]
- Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [pdf]
- Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [pdf]
