Theory Wiki - User contributions [en]

Roll - Productive Failure in a Chemistry Virtual Lab

2010-02-02T01:50:26Z

Idoroll:

Productive Failure in a Chemistry Virtual Laboratory (Roll)
==Summary Table==
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll, David Yaron
|-
| '''Other Contributers''' || Michael Karabinos (Instructional Designer, Carnegie Mellon), Sophia Nussbaum (Course Instructor and Instructional Designer, University of British Columbia)
|-
| '''Study Start Date''' || March, 2010
|-
| '''Study End Date''' || March, 2011
|-
| '''LearnLab Site''' || University of British Columbia, Freshman Laboratory Course
|-
| '''LearnLab Course''' || Chemistry
|-
| '''Number of Students''' || 1100
|-
| '''Total Participant Hours''' || 3300
|-
| '''DataShop''' || Log files of student interactions with virtual lab and other instructional materials.
|}

==Abstract==
This study is occuring in a set of online materials that students complete in preparation for a physical laboratory experience involving the design of a buffer solution. At the beginning of the online activities, students complete a set of activities in the ChemCollective virtual laboratory. In past years, these virtual lab activities have following a direct instruction approach, in which students add acid to a buffered versus unbuffered solution and compare the effects. This study adds an additional condition in which students are asked to create their own buffer solution, with only minimal guidance. While not much guidance is given, the task is designed to [[focusing | attract]] students' attention to deep features of the domain . This is a complex task which is the topic of the following instruction. Students are not expected to succeeed at the task in this initial exploratory phase, but the hypothesis is that by identifying the deep structure of the domain, students would be more likely to learn better from the subsequent instruction. Thus, engaging with the task at the beginning will better prepare them for the formal knowledge which is to follow.

==Background & Significance==
==Glossary==
==Research questions==
Will engagement with an initial exploratory phase of instruction promote student learning, even though the students are likely to fail at meeting the goals of the exploratory activity (designing a buffer solution with a specified pH and buffer capacacity)?
==Independent Variables==
==Dependent Variables==
==Hypothesis==
==Results==
==Explanation==
==Further Information==
This study is currently under design and will be carried out at UBC following the break the university is taking for the winter olympics (data collection begins on March 1, 2010).
===Connections to Other Studies===
===Annotated Bibliography===
===References===
===Future Plans===

Roll - Inquiry

2009-12-04T23:04:43Z

Idoroll: /* References */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* [[invention task|Structured Invention Tasks]]
* [[metacognition|Self-Regulated Learning]]
* [[Scientific Reasoning]]
* [[Inquiry tasks]]

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What scientific reasoning skills are being used and practiced during structured invention tasks?

Step 2: How well do these skills transfer across topics and along time (that is, is there learning of scientific-reasoning skills from one invention task to the next?)

Step 3: What support can best improve performance and learning of scientific reasoning skills?

==Independent Variables==

The first step of the project is an ethnography. Independent variables are the domain, topic, task, # of prior invention tasks. and experience of the group (that is, is this a new group, or does it run together for a long time).

==Dependent Variables==

The first step of this study includes analysis of students' transcripts while working on structured invention tasks.

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

This is part 2 of the [[Roll IPL|IPL]] study previously completed by Roll.

It is also related to the study of Dan Belenky and Tim Nokes about motivational benefits of IPL.

===Annotated Bibliography===

===References===

Kapur, M., & Lee, K. (2009). Designing for productive failure in mathematical problem solving. In Proceedings of the 31st annual conference of the cognitive science society. (pp. 2632-7). Austin, TX: Cognitive Science Society.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T23:04:02Z

Idoroll: /* Future Plans */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* [[invention task|Structured Invention Tasks]]
* [[metacognition|Self-Regulated Learning]]
* [[Scientific Reasoning]]
* [[Inquiry tasks]]

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What scientific reasoning skills are being used and practiced during structured invention tasks?

Step 2: How well do these skills transfer across topics and along time (that is, is there learning of scientific-reasoning skills from one invention task to the next?)

Step 3: What support can best improve performance and learning of scientific reasoning skills?

==Independent Variables==

The first step of the project is an ethnography. Independent variables are the domain, topic, task, # of prior invention tasks. and experience of the group (that is, is this a new group, or does it run together for a long time).

==Dependent Variables==

The first step of this study includes analysis of students' transcripts while working on structured invention tasks.

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

This is part 2 of the [[Roll IPL|IPL]] study previously completed by Roll.

It is also related to the study of Dan Belenky and Tim Nokes about motivational benefits of IPL.

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T23:03:41Z

Idoroll: /* Connections to Other Studies */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* [[invention task|Structured Invention Tasks]]
* [[metacognition|Self-Regulated Learning]]
* [[Scientific Reasoning]]
* [[Inquiry tasks]]

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What scientific reasoning skills are being used and practiced during structured invention tasks?

Step 2: How well do these skills transfer across topics and along time (that is, is there learning of scientific-reasoning skills from one invention task to the next?)

Step 3: What support can best improve performance and learning of scientific reasoning skills?

==Independent Variables==

The first step of the project is an ethnography. Independent variables are the domain, topic, task, # of prior invention tasks. and experience of the group (that is, is this a new group, or does it run together for a long time).

==Dependent Variables==

The first step of this study includes analysis of students' transcripts while working on structured invention tasks.

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

This is part 2 of the [[Roll IPL|IPL]] study previously completed by Roll.

It is also related to the study of Dan Belenky and Tim Nokes about motivational benefits of IPL.

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T23:00:25Z

Idoroll: /* Independent Variables */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* [[invention task|Structured Invention Tasks]]
* [[metacognition|Self-Regulated Learning]]
* [[Scientific Reasoning]]
* [[Inquiry tasks]]

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What scientific reasoning skills are being used and practiced during structured invention tasks?

Step 2: How well do these skills transfer across topics and along time (that is, is there learning of scientific-reasoning skills from one invention task to the next?)

Step 3: What support can best improve performance and learning of scientific reasoning skills?

==Independent Variables==

The first step of the project is an ethnography. Independent variables are the domain, topic, task, # of prior invention tasks. and experience of the group (that is, is this a new group, or does it run together for a long time).

==Dependent Variables==

The first step of this study includes analysis of students' transcripts while working on structured invention tasks.

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:58:49Z

Idoroll: /* Dependent Variables */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* [[invention task|Structured Invention Tasks]]
* [[metacognition|Self-Regulated Learning]]
* [[Scientific Reasoning]]
* [[Inquiry tasks]]

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What scientific reasoning skills are being used and practiced during structured invention tasks?

Step 2: How well do these skills transfer across topics and along time (that is, is there learning of scientific-reasoning skills from one invention task to the next?)

Step 3: What support can best improve performance and learning of scientific reasoning skills?

==Independent Variables==

==Dependent Variables==

The first step of this study includes analysis of students' transcripts while working on structured invention tasks.

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:57:35Z

Idoroll: /* Research questions */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* [[invention task|Structured Invention Tasks]]
* [[metacognition|Self-Regulated Learning]]
* [[Scientific Reasoning]]
* [[Inquiry tasks]]

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What scientific reasoning skills are being used and practiced during structured invention tasks?

Step 2: How well do these skills transfer across topics and along time (that is, is there learning of scientific-reasoning skills from one invention task to the next?)

Step 3: What support can best improve performance and learning of scientific reasoning skills?

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:52:09Z

Idoroll: /* Glossary */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* [[invention task|Structured Invention Tasks]]
* [[metacognition|Self-Regulated Learning]]
* [[Scientific Reasoning]]
* [[Inquiry tasks]]

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:45:46Z

Idoroll: /* Background & Significance */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* Structured Invention Tasks
* Self-Regulated Learning
* Scientific Reasoning
* Inquiry tasks

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:42:09Z

Idoroll: /* Abstract */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What scientific reasoning skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

This project focuses on SRL behavior during scientific inquiry, and relationships between SRL behavior and domain learning and motivation. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

In this project I evaluate whether supporting students’ metacognitive behavior in inquiry tasks helps students acquire better domain and scientific reasoning skills, without reducing the motivational benefits and high agency that students have in inquiry tasks. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

the current project first seeks to identify the SRL skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* Structured Invention Tasks
* Self-Regulated Learning
* Scientific Reasoning
* Inquiry tasks

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:41:33Z

Idoroll: /* Abstract */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What SRL skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

This project focuses on SRL behavior during scientific inquiry, and relationships between SRL behavior and domain learning and motivation. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

In this project I evaluate whether supporting students’ metacognitive behavior in inquiry tasks helps students acquire better domain and scientific reasoning skills, without reducing the motivational benefits and high agency that students have in inquiry tasks. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

the current project first seeks to identify the SRL skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* Structured Invention Tasks
* Self-Regulated Learning
* Scientific Reasoning
* Inquiry tasks

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:35:37Z

Idoroll: /* Abstract */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates SRL behavior within one type of inquiry tasks - structured invention tasks.
The project uses qualitative and quantitative methods to answer three questions:
1. What SRL skills are being practiced during structured invention tasks?
2. How well do these transfer across topics and along time?
3. How can these skills be supported and improved?

==Background & Significance==

This project focuses on SRL behavior during scientific inquiry, and relationships between SRL behavior and domain learning and motivation. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

In this project I evaluate whether supporting students’ metacognitive behavior in inquiry tasks helps students acquire better domain and scientific reasoning skills, without reducing the motivational benefits and high agency that students have in inquiry tasks. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

the current project first seeks to identify the SRL skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* Structured Invention Tasks
* Self-Regulated Learning
* Scientific Reasoning
* Inquiry tasks

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:01:17Z

Idoroll: /* Background & Significance */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

==Background & Significance==

This project focuses on SRL behavior during scientific inquiry, and relationships between SRL behavior and domain learning and motivation. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

In this project I evaluate whether supporting students’ metacognitive behavior in inquiry tasks helps students acquire better domain and scientific reasoning skills, without reducing the motivational benefits and high agency that students have in inquiry tasks. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

the current project first seeks to identify the SRL skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* Structured Invention Tasks
* Self-Regulated Learning
* Scientific Reasoning
* Inquiry tasks

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T22:01:03Z

Idoroll: /* Background & Significance */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

==Background & Significance==

This project focuses on SRL behavior during scientific inquiry, and relationships between SRL behavior and domain learning and motivation. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

In this project I evaluate whether supporting students’ metacognitive behavior in inquiry tasks helps students acquire better domain and scientific reasoning skills, without reducing the motivational benefits and high agency that students have in inquiry tasks. I focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

the current project first seeks to identify the SRL skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, I will investigate the effect of supporting these skills on students' domain and metacognitive learning.

==Glossary==

* Structured Invention Tasks
* Self-Regulated Learning
* Scientific Reasoning
* Inquiry tasks

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T21:52:54Z

Idoroll: /* References */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

==Background & Significance==

This project focuses on SRL behavior during scientific inquiry, and relationships between SRL behavior and domain learning and motivation. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).
This project evaluates whether supporting students’ metacognitive behavior in inquiry tasks helps students acquire better domain and scientific reasoning skills, without reducing the motivational benefits and high agency that students have in inquiry tasks. I focus on the Invention as Preparation for Learning framework (Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In Invention as Preparation for Learning students first attempt to invent a mathematical procedures to evaluate a target property (e.g., variability, probability, etc). Following the invention attempt students receive direct instruction on the canonical procedure, and practice it. Invention as Preparation for learning was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

==Glossary==

* Structured Invention Tasks
* Self-Regulated Learning
* Scientific Reasoning
* Inquiry tasks

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T21:51:07Z

Idoroll: /* Background & Significance */

Roll - Inquiry

2009-12-04T20:13:27Z

Idoroll:

Roll - Inquiry

2009-12-04T19:56:25Z

Idoroll:

Roll - Inquiry

2009-12-04T19:53:57Z

Idoroll: /* Summary Table */

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

= Helping Students Become Better Scientists Using Structured Inquiry Tasks =

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ido Roll
|-
| '''Other Contributers''' || Doug Bonn, James Day
|-
| '''Study Start Date''' || Jan. 1, 2010
|-
| '''Study End Date''' || May. 31, 2010
|-
| '''Site''' || UBC (not a LeanLab site)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = ~200
|-
| '''Total Participant Hours''' || ~1,000.
|-
| '''DataShop''' || no data yet
|}
 

==Abstract==

==Background & Significance==

==Glossary==

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

==Abstract==

==Background & Significance==

==Glossary==

==Research questions==

The project has 3 steps, each of which focuses on a different research question:

Step 1: What SRL skills are being used and practiced during structured invention tasks?

Step 2:

==Independent Variables==

==Dependent Variables==

==Hypothesis==

==Results==

==Explanation==

==Further Information==

===Connections to Other Studies===

===Annotated Bibliography===

===References===

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

===Future Plans===

Spring 2010:
Do an ethnography in in a 1st year physics lab that uses invention tasks as a normal classroom practice.

Roll - Inquiry

2009-12-04T19:50:28Z

Idoroll: New page: = Helping Students Become Better Scientists Using Structured Inquiry Tasks = === Summary Table === {| border="1" cellspacing="0" cellpadding="5" style="text-align: left;" | '''PIs''' || I...

The Help Tutor Roll Aleven McLaren

2009-05-21T04:23:19Z

Idoroll: /* Evaluation of goal 4: Improve future metacognitive behavior */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:help-seeking_behavior.jpg]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

As mentioned earlier, the two units of study 2 were spread apart by one month. We collected data from students' behavior in the months between the two units and following the study. During these months students repeated previous material in preparation for statewide exams.
As seen in the table below, the main effects of the help-seeking environment persisted even once students moved on to working with the native Cognitive Tutor! Overall, students who received help-seeking support during the study took more time for their actions following the study, especially for reading hints - their hint reading time before asking for additional hint is longer by almost one effect size in the month following the study (12 vs. 8 seconds, t(44)=3.0, p<.01). Also, students who were in the Help condition did not drill down through the hints as often, though this effect is only marginal significant (average hint level: 2.2 vs. 2.6, t(51)=1.9, p=.06). These effects are more consistently significant after both units, suggesting that having the study stretched across two units indeed helped students better acquire the domain-independent help-seeking skills.

[[Image:four months of HT.jpg]]

In addition, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
* The Help Seeking Support Environment intervened appropriate actions during the learning process.
* Students improved their help-seeking behavior while working with the system.
* Students demonstrated improved help-seeking behavior even once support was removed, after working with the Help Tutor along two months and two different topics.
* Furthermore, students acquired better help-seeking declarative knowledge.
* However, students' domain learning did not improve.

=== Explanation ===

==== 1 Assessing help-seeking behavior ====
These studies put forward several direct measures of help seeking. Most notably, we found that the online help-seeking model is able to capture faulty help-seeking behavior in a manner that is consistent with domain learning. The operational nature of the model (i.e., the fact that it can be run on a computer) puts it in unique position, compared to other models of help-seeking behavior. Furthermore, formative assessment using the model is done in a transparent manner that does not interrupt the learning process. These qualities of the model enable us to do a micro-genetic (moment-by-moment) analysis of help-seeking behavior over extended periods of time, quite unique in the literature on help seeking and on self-regulated learning. In addition, the detailed evaluation of students’ actions allows us to adapt the learning process to the learner in novel ways – that is, adapt not only to students’ domain knowledge and behavior, but also to their metacognitive knowledge and behavior.
Students’ online behavior as assessed by the help-seeking model also correlated with paper-and-pencil measures of help seeking knowledge and behavior. This result provides some support that the measures capture a metacognitive behavior that is domain and environment independent in nature. A combination of these instruments with domain learning measures can be used to investigate the different factors affecting learning behaviors and outcomes.
==== 2. Improving help-seeking behavior and knowledge ====
Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.
==== 3 Using metacognitive improvement to improve domain learning ====
The two studies did not find an effect of the improved help-seeking behavior on domain learning. One possible explanation might be that the help-seeking model focuses on the wrong actions, although the negative correlation with learning across studies suggests otherwise. Alternatively, perhaps the improvement in help-seeking behavior was not sufficient to measurably impact domain-level learning. However, we would expect to see at least a trend in the test scores, rather than the virtual tie we observed across the three units.
A third explanation (see paper #8 by authors) is that poor help-seeking behavior may be more a consequence of poor motivation toward domain learning than it is a cause. Students who do not care to learn geometry may exhibit poor help seeking behaviors and learn less, thus yielding a correlation. However, in this case, poor help seeking behavior may not reflect a lack of help-seeking skill per se, but a lack of motivation to deeply engage in help seeking and associated deep learning strategies (like attempting to encode instructional material in terms of deeper domain-relevant features rather than more superficial perceptual features). Indeed Help condition students changed their help-seeking behaviors and appeared to do so in a lasting way, but perhaps associated changes in other deep learning strategies are needed before clear effects on domain learning can be observed.
To elaborate, consider that the help-seeking model focuses on how deliberately students seek help, but does not evaluate the content of the help or how it is being used. It makes sure that students take enough time to read the hint, but does not assess how this information is being processed. One of the implicit assumptions of this line of research is that students can learn from given explanations in contextual hints. However, it may be that this process is more challenging. For example, Renkl showed that at times instructional explanations actually hinder students’ tendency to self-explain and thus learn (Renkl, 2002). In other words, perhaps students do not learn enough from help not only because of how they obtain it, but also because of how they process it. Reevaluating the existing help-seeking literature supports this hypothesis. Very few experiments have actively manipulated help seeking in ITSs to date. Those who manipulated the content of the help or students’ reaction to it often found that interventions that increase reflection yield better learning: Dutke and Reimer (2000) found that principle-based hints are better than operative ones; Ringenberg and VanLehn (2006) found that analogous solved examples may be better than conventional hints; Schworm and Renkl (2002) found that deep reflection questions caused more learning compared with conventional hints; and Baker et al. (2006) showed that auxiliary exercises for students who misuse hints help them learn better. At the same time, Baker (2006) found that reducing help abuse (and other gaming behaviors) may not contribute to learning gains by itself, a similar result to the ones presented in this paper.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

The Help Tutor Roll Aleven McLaren

2009-05-21T04:22:09Z

Idoroll: /* Further Information */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:help-seeking_behavior.jpg]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:four months of HT.jpg]]

In addition, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
* The Help Seeking Support Environment intervened appropriate actions during the learning process.
* Students improved their help-seeking behavior while working with the system.
* Students demonstrated improved help-seeking behavior even once support was removed, after working with the Help Tutor along two months and two different topics.
* Furthermore, students acquired better help-seeking declarative knowledge.
* However, students' domain learning did not improve.

=== Explanation ===

==== 1 Assessing help-seeking behavior ====
These studies put forward several direct measures of help seeking. Most notably, we found that the online help-seeking model is able to capture faulty help-seeking behavior in a manner that is consistent with domain learning. The operational nature of the model (i.e., the fact that it can be run on a computer) puts it in unique position, compared to other models of help-seeking behavior. Furthermore, formative assessment using the model is done in a transparent manner that does not interrupt the learning process. These qualities of the model enable us to do a micro-genetic (moment-by-moment) analysis of help-seeking behavior over extended periods of time, quite unique in the literature on help seeking and on self-regulated learning. In addition, the detailed evaluation of students’ actions allows us to adapt the learning process to the learner in novel ways – that is, adapt not only to students’ domain knowledge and behavior, but also to their metacognitive knowledge and behavior.
Students’ online behavior as assessed by the help-seeking model also correlated with paper-and-pencil measures of help seeking knowledge and behavior. This result provides some support that the measures capture a metacognitive behavior that is domain and environment independent in nature. A combination of these instruments with domain learning measures can be used to investigate the different factors affecting learning behaviors and outcomes.
==== 2. Improving help-seeking behavior and knowledge ====
Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.
==== 3 Using metacognitive improvement to improve domain learning ====
The two studies did not find an effect of the improved help-seeking behavior on domain learning. One possible explanation might be that the help-seeking model focuses on the wrong actions, although the negative correlation with learning across studies suggests otherwise. Alternatively, perhaps the improvement in help-seeking behavior was not sufficient to measurably impact domain-level learning. However, we would expect to see at least a trend in the test scores, rather than the virtual tie we observed across the three units.
A third explanation (see paper #8 by authors) is that poor help-seeking behavior may be more a consequence of poor motivation toward domain learning than it is a cause. Students who do not care to learn geometry may exhibit poor help seeking behaviors and learn less, thus yielding a correlation. However, in this case, poor help seeking behavior may not reflect a lack of help-seeking skill per se, but a lack of motivation to deeply engage in help seeking and associated deep learning strategies (like attempting to encode instructional material in terms of deeper domain-relevant features rather than more superficial perceptual features). Indeed Help condition students changed their help-seeking behaviors and appeared to do so in a lasting way, but perhaps associated changes in other deep learning strategies are needed before clear effects on domain learning can be observed.
To elaborate, consider that the help-seeking model focuses on how deliberately students seek help, but does not evaluate the content of the help or how it is being used. It makes sure that students take enough time to read the hint, but does not assess how this information is being processed. One of the implicit assumptions of this line of research is that students can learn from given explanations in contextual hints. However, it may be that this process is more challenging. For example, Renkl showed that at times instructional explanations actually hinder students’ tendency to self-explain and thus learn (Renkl, 2002). In other words, perhaps students do not learn enough from help not only because of how they obtain it, but also because of how they process it. Reevaluating the existing help-seeking literature supports this hypothesis. Very few experiments have actively manipulated help seeking in ITSs to date. Those who manipulated the content of the help or students’ reaction to it often found that interventions that increase reflection yield better learning: Dutke and Reimer (2000) found that principle-based hints are better than operative ones; Ringenberg and VanLehn (2006) found that analogous solved examples may be better than conventional hints; Schworm and Renkl (2002) found that deep reflection questions caused more learning compared with conventional hints; and Baker et al. (2006) showed that auxiliary exercises for students who misuse hints help them learn better. At the same time, Baker (2006) found that reducing help abuse (and other gaming behaviors) may not contribute to learning gains by itself, a similar result to the ones presented in this paper.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

The Help Tutor Roll Aleven McLaren

2009-05-21T04:21:49Z

Idoroll: /* Explanation */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:help-seeking_behavior.jpg]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:four months of HT.jpg]]

In addition, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
* The Help Seeking Support Environment intervened appropriate actions during the learning process.
* Students improved their help-seeking behavior while working with the system.
* Students demonstrated improved help-seeking behavior even once support was removed, after working with the Help Tutor along two months and two different topics.
* Furthermore, students acquired better help-seeking declarative knowledge.
* However, students' domain learning did not improve.

=== Explanation ===

==== 1 Assessing help-seeking behavior ====
These studies put forward several direct measures of help seeking. Most notably, we found that the online help-seeking model is able to capture faulty help-seeking behavior in a manner that is consistent with domain learning. The operational nature of the model (i.e., the fact that it can be run on a computer) puts it in unique position, compared to other models of help-seeking behavior. Furthermore, formative assessment using the model is done in a transparent manner that does not interrupt the learning process. These qualities of the model enable us to do a micro-genetic (moment-by-moment) analysis of help-seeking behavior over extended periods of time, quite unique in the literature on help seeking and on self-regulated learning. In addition, the detailed evaluation of students’ actions allows us to adapt the learning process to the learner in novel ways – that is, adapt not only to students’ domain knowledge and behavior, but also to their metacognitive knowledge and behavior.
Students’ online behavior as assessed by the help-seeking model also correlated with paper-and-pencil measures of help seeking knowledge and behavior. This result provides some support that the measures capture a metacognitive behavior that is domain and environment independent in nature. A combination of these instruments with domain learning measures can be used to investigate the different factors affecting learning behaviors and outcomes.
==== 2. Improving help-seeking behavior and knowledge ====
Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.
==== 3 Using metacognitive improvement to improve domain learning ====
The two studies did not find an effect of the improved help-seeking behavior on domain learning. One possible explanation might be that the help-seeking model focuses on the wrong actions, although the negative correlation with learning across studies suggests otherwise. Alternatively, perhaps the improvement in help-seeking behavior was not sufficient to measurably impact domain-level learning. However, we would expect to see at least a trend in the test scores, rather than the virtual tie we observed across the three units.
A third explanation (see paper #8 by authors) is that poor help-seeking behavior may be more a consequence of poor motivation toward domain learning than it is a cause. Students who do not care to learn geometry may exhibit poor help seeking behaviors and learn less, thus yielding a correlation. However, in this case, poor help seeking behavior may not reflect a lack of help-seeking skill per se, but a lack of motivation to deeply engage in help seeking and associated deep learning strategies (like attempting to encode instructional material in terms of deeper domain-relevant features rather than more superficial perceptual features). Indeed Help condition students changed their help-seeking behaviors and appeared to do so in a lasting way, but perhaps associated changes in other deep learning strategies are needed before clear effects on domain learning can be observed.
To elaborate, consider that the help-seeking model focuses on how deliberately students seek help, but does not evaluate the content of the help or how it is being used. It makes sure that students take enough time to read the hint, but does not assess how this information is being processed. One of the implicit assumptions of this line of research is that students can learn from given explanations in contextual hints. However, it may be that this process is more challenging. For example, Renkl showed that at times instructional explanations actually hinder students’ tendency to self-explain and thus learn (Renkl, 2002). In other words, perhaps students do not learn enough from help not only because of how they obtain it, but also because of how they process it. Reevaluating the existing help-seeking literature supports this hypothesis. Very few experiments have actively manipulated help seeking in ITSs to date. Those who manipulated the content of the help or students’ reaction to it often found that interventions that increase reflection yield better learning: Dutke and Reimer (2000) found that principle-based hints are better than operative ones; Ringenberg and VanLehn (2006) found that analogous solved examples may be better than conventional hints; Schworm and Renkl (2002) found that deep reflection questions caused more learning compared with conventional hints; and Baker et al. (2006) showed that auxiliary exercises for students who misuse hints help them learn better. At the same time, Baker (2006) found that reducing help abuse (and other gaming behaviors) may not contribute to learning gains by itself, a similar result to the ones presented in this paper.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

The Help Tutor Roll Aleven McLaren

2009-05-21T04:20:24Z

Idoroll: /* Summary of results */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:help-seeking_behavior.jpg]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:four months of HT.jpg]]

In addition, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
* The Help Seeking Support Environment intervened appropriate actions during the learning process.
* Students improved their help-seeking behavior while working with the system.
* Students demonstrated improved help-seeking behavior even once support was removed, after working with the Help Tutor along two months and two different topics.
* Furthermore, students acquired better help-seeking declarative knowledge.
* However, students' domain learning did not improve.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

The Help Tutor Roll Aleven McLaren

2009-05-21T04:18:23Z

Idoroll: /* Evaluation of goal 4: Improve future metacognitive behavior */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:help-seeking_behavior.jpg]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:four months of HT.jpg]]

In addition, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

The Help Tutor Roll Aleven McLaren

2009-05-21T04:16:55Z

Idoroll: Undo revision 9497 by Idoroll (Talk)

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:help-seeking_behavior.jpg]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

To evaluate whether the effect of the help-seeking curriculum persists beyond the tutored environment, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

The Help Tutor Roll Aleven McLaren

2009-05-21T04:16:23Z

Idoroll:

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

[[Image:help-seeking_behavior.jpg]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

To evaluate whether the effect of the help-seeking curriculum persists beyond the tutored environment, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

The Help Tutor Roll Aleven McLaren

2009-05-21T04:14:48Z

Idoroll: /* Evaluation of goal 2: Improve metacognitive behavior */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

[[Image:four months of HT.jpg]]

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

To evaluate whether the effect of the help-seeking curriculum persists beyond the tutored environment, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

File:Four months of HT.jpg

2009-05-21T04:14:22Z

Idoroll:

The Help Tutor Roll Aleven McLaren

2009-05-21T04:12:43Z

Idoroll: /* Evaluation of goal 2: Improve metacognitive behavior */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren, Ken Koedinger

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Students’ hint usage improved significantly while working with the Help Tutor across several measures, most notably, help-seeking error rate, frequency of asking for bottom-out hints, and hint reading time. Study 2 also shows that these aspects of improvement persist even beyond the scope of the Help Tutor. These improvements consistently reached significance only after an extended period of time of working with the Help Tutor (i.e., after the second part of the study). We hypothesize that since the Help Tutor feedback appeared in two different areas of geometry learning, students could more easily acquire the domain-independent help-seeking skills and thus transfer them better to the other subject matter areas addressed between and after the two in which the Help Tutor was used.
The addition of the help-seeking instruction and self-assessment episodes in Study 2 led to improvements in students’ conceptual help seeking knowledge, time to read hints, and an apparent improvement in hint-to-error ratio. The fact that students were more likely to ask for hints rather than committing more errors provide behavioral evidence that self-assessment instruction provided by both the Self-Assessment tutor and by certain messages in the Help Tutor appear to lead students to be more aware of their need for help, reinforcing the causal link between self-assessment and strategic help seeking (Tobias & Everson, 2002).
Although students’ hint usage improved, no major improvements in the deliberateness of students’ solution attempts were found (besides that indicated by the differences in hint-to-error ratio). It may be that the Help Tutor did not support this aspect of learning well enough. An alternative explanation is that as long as students attempted to solve problems (whether successfully or not) they were too occupied by the problem-solving attempts and thus did not pay attention (and cognitive resources) to the Help Tutor until they reached an impasse that required them to ask for help. This may be an outcome of our design decision to give feedback in the context of domain learning.

[[Image:four months of HT.jpg]]

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

To evaluate whether the effect of the help-seeking curriculum persists beyond the tutored environment, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. International Conference on Artificial Intelligence in Education, , 203-10. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition - applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2). [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

Roll IPL

2009-05-21T03:58:37Z

Idoroll: /* Findings */

== Invention as preparation for learning ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

Can invention activities prepare students to better learn from subsequent instruction, compared with instruction-and-practice only?

=== Meta-data ===

PI's: Ido Roll, Vincent Aleven, Dan Schwartz, Ken Koedinger

Other Contributers: David Klahr
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 4/2007 || 4/2007 || North Hills || 20 || 40 || No, paper-and-pencil only
|-
| '''2''' || 9/2007 || 12/2007 || Community Day || 4 || 48 || No, paper-and-pencil only
|-
| '''3''' || 4/2008 || 5/2008 || Steel Valley || 125 || 900 || No, paper-and-pencil only
|-
| '''4''' || 4/2009 || 5/2009 || Steel Valley || 140 || 560 || Some of it in DataShop, the rest is getting there
|-
|}

=== Abstract ===

The [[assistance dilemma]] asks what form of assistance is most appropriate for different stages of learning. While direct instruction and practice have been shown to be efficient for novices, students often acquire shallow knowledge components and lack robust understanding. Some evidence suggests that invention using contrasting cases, prior to instruction and practice, can accelerate future learning, compared with instruction and practice alone (Schwartz & Martin, 2004).
The invention process as described in Schwartz & Martin (2004) includes the following stages: Design (of a mathematical model to solve a class of problems); calculation (of the solution based on the model); evaluation (of its correctness); and debugging (of the faulty model). Notably, most students fail to invent mathematically valid models, so the goal is not for students to discover the correct solution. At the same time, students do make models that capture deep features of the class of problems, which prepares them to learn and understand the significance of expert solutions for handling such situations. Following the invention, students receive instruction on the expert solution (that is, formulas) and practice it. This procedure is based on the hypothesis that students’ own inventions, together with subsequent instruction, are sources for coordinative learning. By attempting to create a model that correctly distinguishes the “contrasting cases” (carefully selected instances within a class of problems) students notice (and to some degree invent) the problem features that an adequate model must take into account, and they attend to them during subsequent instruction. However, alternative explanations for the effectiveness of the IPL process are possible, with different instructional implications. A “debugging hypothesis” suggests that evaluation and debugging of pre-designed models are sufficient to promote future learning by directing students’ attention to the short-comings of the designed models, and thus to the deep features of the domain. Alternatively, an “unfinished goals” hypothesis suggests that the effect is caused by students reaching impasses during invention. According to this hypothesis, calculation-evaluation are sufficient for preparing for future learning.
We propose to investigate this in a series of ablation studies with the goal of better defining the invention process and identifying the cognitive processes involved. This includes a combination of in-vivo and lab studies within the Algebra LearnLab, contributing to the Coordinative Learning theoretical framework. Following the ablation studies we plan to implement the procedure in a Cognitive Tutor, which will be evaluated in a lab study. This will allow us to better operationalize the process, do a micro-genetic analysis of it, and identify productive patterns of learning trajectories using log mining.

=== Background and Significance ===

One of the main challenges of education is to help students reach meaningful and robust learning. The assistance dilemma raises the question of what form (and ‘amount’) of assistance are most effective with different learners in different stages of the learning process (Koedinger & Aleven, in press). Instruction followed by practice is known to be very efficient for teaching novices (e.g., Koedinger, Anderson, Hadley & Mark, 1997); yet, students often acquire shallow procedural skills, and fail to acquire conceptual understanding (Aleven & Koedinger, 2002). This can be attributed, at least in part, to students using superficial features and not encoding the deep features of the domain (Chi, Feltovich & Glaser, 1981).
One approach to getting students to attend and encode the deep features is to add an invention phase prior to instruction. Invention as preparation for leaning (IPL) was shown to help students better cope with novel situations that require learning (Schwartz & Martin, 2004; Sears, 2006). In this process students are presented with a dilemma in the form of contrasting cases, and attempt to invent a mathematical model to resolve this dilemma. For example, Figure 1 shows four possible pitching machines. Students are asked to invent a method that will allow them to pick the most reliable machine. The concept of contrasting cases comes from the perceptual learning literature, since these cases, when appropriately designed, emphasize differences in the deep structure of the examples (Gibson & Gibson, 1955). The invention process includes designing a model, applying it to the given set of contrasting cases, evaluating the result, and debugging the model. This iterative process is very similar to the debugging process as described by Klahr and Carver (1988; Figure 2). Unlike other inquiry-based manipulations (cf. Lehrer et al., 2001; de Jong & van Joolingen, 1998), the goal of the IPL process is not for students to discover the correct model, but to prepare them for subsequent instruction. During the instruction students share their models, critic their peers’ models, and learn the expert solutions (A similar classroom critic process was shown to be effective by White & Frederiksen, 1998). Preparation for learning from the instruction is evaluated using accelerated future learning assessment. The accelerated future learning assessment includes an embedded instruction in the test in the form of solved example. Schwartz (2004) found that only students who invented prior to the test were able to take advantage of that instruction in order to solve novel problem, while students who practiced a given visual method prior to the test did not take advantage of the embedded learning resource and thus could not solve the target problem. This shows that the IPL process has a positive effect on students’ ability to independently learn from the solved examples. In the case of the contrasting cases given in Figure 1, subsequent instruction will introduce the students with the notion (and formulas) of variance.
While the invention group was superior to instruction-and-practice group on accelerated future learning measure, there was no direct comparison of normal or transfer measures between invention and instruction-and-practice conditions (though invention students showed pre-to-post gains, and were shown to outperformed college students). Also, it is not yet clear how robust this pedagogy is and what its key features are.

Example for contrasting cases (topic - variability)

[[Image:contrasting cases.jpg]]

The overall IPL process:

[[Image:IPL process.jpg]]

=== Glossary ===
See [[:Category:IPL|IPL Glossary]]

=== Research questions ===

# What is the overall effect of [[Invention task]]s on students domain knowledge, sense-making skills, and motivation, compared with [[direct instruction]]?
# What elements of invention contribute to that effect? What cognitive processes do they drive? In what ways does knowledge acquired following invention differ from knowledge acquired in direct instruction alone?
# Can the IPL process be scaled-up using technology?

In addition, the project makes the following contributions:
# It compares different measures of [[robust learning]], in order to understand what aspect of knowledge can be assessed using what type of measure.

=== Independent Variables ===

Different studies manipulated different stages of the [[Invention task]]:
- Observation (a.k.a. comparative reasoning): comparing contrasting cases that vary along deep features, with regard to target concepts
- Generative reasoning: designing novel mathematical procedures to compare the contrasting cases with regard to the target concept
- Evaluation: Evaluation of the models

=== Dependent variables ===

Domain knowledge (in increasing 'distance' from instruction):
* Normal measures
* Transfer measures
* New strategy items (with learning resource)
* New strategy items (without learning resource)

[[Image:IPL measures.jpg]]

Motivation and affect:
* Behavioral measure: % of students who kept working during breaks
* Self reports

=== Hypothesis ===

One hypothesis argues that generative reasoning (in the form of symbolic invention) is necessary to improve encoding of subsequent instruction. First, generative reasoning facilitates a process in which students express their prior ideas, identify their shortcomings, and refine their mental models, thus enabling conceptual change (Smith, diSessa, & Roschelle, 1994). For example, the self-explanation literature shows that asking students to explain their errors facilitates conceptual shift (c.f., Siegler, 2002).
By attempting to invent and understand how different symbolic procedures succeed (or fail) to capture the differences between the contrasting cases, students also acquire a more cohesive and integrated understanding of the deep features of the domain. The importance of the symbolic nature of the process was demonstrated by Schwartz, Martin, and Pfaffman (2005), who asked students to reason verbally or mathematically about the balance beam problem. All students noticed the deep features of the balance beam domain - distance and weight. However, only students who reasoned mathematically were able to reconcile the two dimensions to a single representation. Interestingly, students’ thinking evolved even though their solutions were not complete, similar to the IPL effect.
Lastly, the generative reasoning process may help students understand the function of the different components of the procedure (for example, dividing by N controls for sample size). Thus, students may encode the subsequent instruction by function and not merely by procedure. Functional mental models were previously shown to lead to better adaptation of knowledge (Kieras & Bovair, 1984). Hatano and Inagaki (1986) describe a similar process in which developing mental models of how procedures interact with empirical knowledge helps students acquire conceptual understanding of the domain.
An alternative hypothesis argues that comparative reasoning is sufficient to achieve the learning benefits of IPL. According to this hypothesis, the benefits of invention stem from noticing and encoding the deep features of the domain. The comparative reasoning activity achieves that benefit by asking students to compare contrasting cases that differ with respect to their deep features. (Bransford & Schwartz, 2001). This qualitative analysis helps students set requirements for a valid model and thus acquire a better understanding (even if implicit) of the target concepts. Furthermore, according to this hypothesis, not only does the symbolic invention not contribute to future learning, it may waste students’ time (and thus reduce efficiency) or impose excessive cognitive load (Kirschner, Sweller & Clark, 2006).
A second research question addressed by our current study examines the effect of IPL on the flexibility of students’ knowledge. We follow a distinction made by McDaniel and Schlager (1990) between transfer problems that require the application of a learned strategy (conventional transfer problems) and transfer problems that require the generation of a new strategy. McDaniel and Schlager found that while discovery tasks improve students’ performance on the latter, they have no effect on conventional transfer problems. Schwartz and Martin (2004) add a twist to these results. They found that IPL improves students’ ability to solve new-strategy problems as long as they are provided with instruction on how to do so. To further investigate the effect of IPL on knowledge flexibility, we evaluate students’ ability to independently solve new-strategy problems and encode new-strategy instructions. Our hypothesis, as supported by McDaniel and Schlager (1990), is that students who are engaged in IPL will acquire more flexible knowledge and thus will demonstrate better performance on new-strategy items. At the same time they will not show better ability to use existing strategies in novel contexts (conventional transfer items). Furthermore, following the findings of Schwartz and Martin (2004), we hypothesize that the effect of IPL will be mainly on encoding new-strategy instructions.

==== Instructional Principles ====
See [[IPL Instructional Principles]]

=== Findings ===

Domain knowledge:

[[Image:test results.jpg]]

* IPL students in advanced classes were more capable of solving new strategy items without learning resource. In fact, in the absence of a learning resource, direct instruction students performed at floor, while IPL students performed as well as with the source.
* This effect holds when controlling for simple domain knowledge (performance on normal items in the same test).
* This was found in multiple new-strategy items. However, all results were found in a single topic (central tendency and graphing). The single test item on the topic of variability failed to capture differences between conditions.

Motivation:
* IPL students reported to have benefited more (marginally significant: F=3.3, p<.07)
* There was a significant interaction between condition and test anxiety. Text anxiety was assessed using the MSLQ (Pintrich 1999) before the study had begun. Students who reported to have higher test anxiety also reported to have benefited more from IPL instruction compared to high-anxiety students in the no design condition.
* IPL students stayed more often in class to work during breaks (IPL: 16% No Design: 3%).
* Furthermore, they did so during invention activities and not show-and-practice activities, suggesting that it is the activities that are motivating (IPL activities: 25%; show-and-practice activities: 7%)

[[Image:motivational results.jpg]]

=== Explanation ===
Regarding our first research question, we found that generative reasoning (on top of comparative reasoning) had a positive effect on students’ ability to solve new-strategy problems with no learning resource in the advanced classes. At the same time, as hypothesized, it had a marginal effect on normal or conventional transfer items. These results are interesting especially since Full IPL students had approximately half the time for instruction and practice compared with their No Design counterparts.
Regarding the second research question, which dealt with students’ knowledge flexibility, we found that in the advanced classes, students who designed novel methods during IPL were more capable of solving problems that require the use of novel strategies. This finding echoes the effect found by McDaniel and Schlager (1990). Interestingly, the effect of IPL on new-strategy items with no resources holds even when controlling for performance on normal items on the same test. Thus, this effect can probably not be attributed to more domain knowledge. Instead, it is likely the outcome of a different encoding of domain knowledge, in a manner that is not reflected in normal or transfer items.
On further scrutiny, students in both conditions did equally well on all tasks for which they received some form of instruction - whether in class (on normal and conventional transfer items) or embedded in the test (on new-strategy items with embedded learning resources). Regarding the latter, it seems that Full IPL students did not need the additional instruction whereas No Design students did not manage to solve the new-strategy problems without it. The performance of Full IPL students on new-strategy items remained virtually the same even in the absence of embedded instruction. This finding is at odds with earlier findings by Schwartz and Taylor (2004) who found that IPL improves students’ ability to encode future instruction but not solve novel problems without additional instruction. One explanation for the discrepancy between the studies is that the control group in Schwartz and Taylor (2004) did not engage in comparative reasoning. Therefore, it may be that the comparative reasoning stage helped students in our study to encode the novel instruction.
An alternative explanation examines these results in terms of ‘distance’ from original classroom instruction. It may be that the embedded instruction on the first topic in our study was close to the classroom material, and thus simple enough for all students to encode. In contrast, the embedded learning resource in the study described by Schwartz and Martin (2004) was sufficiently far from the classroom instruction. Therefore, only IPL students, who had acquired more flexible knowledge, could learn from it and apply the acquired knowledge successfully. This explanation further suggests that in the absence of additional instruction, only Full IPL students in our study could make the leap and answer the target new-strategy items.
While this argument explains performance on new-strategy items (with or without instruction) in terms of distance from classroom instruction, it does not explain what factors determine this distance. What makes some items ‘closer’ than others? What prepared Full IPL students for improved performance on some items but not on others?
Students may grapple with many challenges during the invention phase, many of which do not receive attention during classroom instruction. Students who invent are exposed to various challenges by virtue of attempting to invent general valid methods. We hypothesize that students use knowledge acquired during these experiences when later integrating new-strategy tasks into their existing body of knowledge. For example, the post-tests in this study included three new-strategy items, requiring the following new strategies: (1) comparing multiple datasets in a single representation; (2) representing data in unconventional intervals; and (3) finding the ratio between variability and average in order to account for differences in magnitude. These topics were not covered during classroom instruction. However, when we analyzed students’ inventions, we noticed that many inventions included features that could prepare students to expand the instructed knowledge and invent the first two strategies (see Figure 3). Subsequently, Full IPL students demonstrated better performance on the relevant new-strategy items. At the same time, no student attempted during invention to compare datasets with different magnitudes. Correspondingly, Full IPL students did not exhibit better performance on this new-strategy item.

=== Connections===

* Tim Nokes's study

=== The Invention Lab ===
In addition to paper and pencil studies, we have created the Invention Lab.

The Invention Lab is an intelligent tutoring system for IPL. To give intelligent feedback, it uses two models:
* A meta-cognitive model of the invention process
* A cognitive model of the main concepts in the domain

[[Image:iLab.jpg]]

=== Annotated bibliography ===

[[Category:Empirical Study]]
[[Category:Protected]]

File:ILab.jpg

2009-05-21T03:56:28Z

Idoroll:

Roll IPL

2009-05-21T03:55:58Z

Idoroll: /* Further Information */

== Invention as preparation for learning ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

Can invention activities prepare students to better learn from subsequent instruction, compared with instruction-and-practice only?

=== Meta-data ===

PI's: Ido Roll, Vincent Aleven, Dan Schwartz, Ken Koedinger

Other Contributers: David Klahr
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 4/2007 || 4/2007 || North Hills || 20 || 40 || No, paper-and-pencil only
|-
| '''2''' || 9/2007 || 12/2007 || Community Day || 4 || 48 || No, paper-and-pencil only
|-
| '''3''' || 4/2008 || 5/2008 || Steel Valley || 125 || 900 || No, paper-and-pencil only
|-
| '''4''' || 4/2009 || 5/2009 || Steel Valley || 140 || 560 || Some of it in DataShop, the rest is getting there
|-
|}

=== Abstract ===

The [[assistance dilemma]] asks what form of assistance is most appropriate for different stages of learning. While direct instruction and practice have been shown to be efficient for novices, students often acquire shallow knowledge components and lack robust understanding. Some evidence suggests that invention using contrasting cases, prior to instruction and practice, can accelerate future learning, compared with instruction and practice alone (Schwartz & Martin, 2004).
The invention process as described in Schwartz & Martin (2004) includes the following stages: Design (of a mathematical model to solve a class of problems); calculation (of the solution based on the model); evaluation (of its correctness); and debugging (of the faulty model). Notably, most students fail to invent mathematically valid models, so the goal is not for students to discover the correct solution. At the same time, students do make models that capture deep features of the class of problems, which prepares them to learn and understand the significance of expert solutions for handling such situations. Following the invention, students receive instruction on the expert solution (that is, formulas) and practice it. This procedure is based on the hypothesis that students’ own inventions, together with subsequent instruction, are sources for coordinative learning. By attempting to create a model that correctly distinguishes the “contrasting cases” (carefully selected instances within a class of problems) students notice (and to some degree invent) the problem features that an adequate model must take into account, and they attend to them during subsequent instruction. However, alternative explanations for the effectiveness of the IPL process are possible, with different instructional implications. A “debugging hypothesis” suggests that evaluation and debugging of pre-designed models are sufficient to promote future learning by directing students’ attention to the short-comings of the designed models, and thus to the deep features of the domain. Alternatively, an “unfinished goals” hypothesis suggests that the effect is caused by students reaching impasses during invention. According to this hypothesis, calculation-evaluation are sufficient for preparing for future learning.
We propose to investigate this in a series of ablation studies with the goal of better defining the invention process and identifying the cognitive processes involved. This includes a combination of in-vivo and lab studies within the Algebra LearnLab, contributing to the Coordinative Learning theoretical framework. Following the ablation studies we plan to implement the procedure in a Cognitive Tutor, which will be evaluated in a lab study. This will allow us to better operationalize the process, do a micro-genetic analysis of it, and identify productive patterns of learning trajectories using log mining.

=== Background and Significance ===

One of the main challenges of education is to help students reach meaningful and robust learning. The assistance dilemma raises the question of what form (and ‘amount’) of assistance are most effective with different learners in different stages of the learning process (Koedinger & Aleven, in press). Instruction followed by practice is known to be very efficient for teaching novices (e.g., Koedinger, Anderson, Hadley & Mark, 1997); yet, students often acquire shallow procedural skills, and fail to acquire conceptual understanding (Aleven & Koedinger, 2002). This can be attributed, at least in part, to students using superficial features and not encoding the deep features of the domain (Chi, Feltovich & Glaser, 1981).
One approach to getting students to attend and encode the deep features is to add an invention phase prior to instruction. Invention as preparation for leaning (IPL) was shown to help students better cope with novel situations that require learning (Schwartz & Martin, 2004; Sears, 2006). In this process students are presented with a dilemma in the form of contrasting cases, and attempt to invent a mathematical model to resolve this dilemma. For example, Figure 1 shows four possible pitching machines. Students are asked to invent a method that will allow them to pick the most reliable machine. The concept of contrasting cases comes from the perceptual learning literature, since these cases, when appropriately designed, emphasize differences in the deep structure of the examples (Gibson & Gibson, 1955). The invention process includes designing a model, applying it to the given set of contrasting cases, evaluating the result, and debugging the model. This iterative process is very similar to the debugging process as described by Klahr and Carver (1988; Figure 2). Unlike other inquiry-based manipulations (cf. Lehrer et al., 2001; de Jong & van Joolingen, 1998), the goal of the IPL process is not for students to discover the correct model, but to prepare them for subsequent instruction. During the instruction students share their models, critic their peers’ models, and learn the expert solutions (A similar classroom critic process was shown to be effective by White & Frederiksen, 1998). Preparation for learning from the instruction is evaluated using accelerated future learning assessment. The accelerated future learning assessment includes an embedded instruction in the test in the form of solved example. Schwartz (2004) found that only students who invented prior to the test were able to take advantage of that instruction in order to solve novel problem, while students who practiced a given visual method prior to the test did not take advantage of the embedded learning resource and thus could not solve the target problem. This shows that the IPL process has a positive effect on students’ ability to independently learn from the solved examples. In the case of the contrasting cases given in Figure 1, subsequent instruction will introduce the students with the notion (and formulas) of variance.
While the invention group was superior to instruction-and-practice group on accelerated future learning measure, there was no direct comparison of normal or transfer measures between invention and instruction-and-practice conditions (though invention students showed pre-to-post gains, and were shown to outperformed college students). Also, it is not yet clear how robust this pedagogy is and what its key features are.

Example for contrasting cases (topic - variability)

[[Image:contrasting cases.jpg]]

The overall IPL process:

[[Image:IPL process.jpg]]

=== Glossary ===
See [[:Category:IPL|IPL Glossary]]

=== Research questions ===

# What is the overall effect of [[Invention task]]s on students domain knowledge, sense-making skills, and motivation, compared with [[direct instruction]]?
# What elements of invention contribute to that effect? What cognitive processes do they drive? In what ways does knowledge acquired following invention differ from knowledge acquired in direct instruction alone?
# Can the IPL process be scaled-up using technology?

In addition, the project makes the following contributions:
# It compares different measures of [[robust learning]], in order to understand what aspect of knowledge can be assessed using what type of measure.

=== Independent Variables ===

Different studies manipulated different stages of the [[Invention task]]:
- Observation (a.k.a. comparative reasoning): comparing contrasting cases that vary along deep features, with regard to target concepts
- Generative reasoning: designing novel mathematical procedures to compare the contrasting cases with regard to the target concept
- Evaluation: Evaluation of the models

=== Dependent variables ===

Domain knowledge (in increasing 'distance' from instruction):
* Normal measures
* Transfer measures
* New strategy items (with learning resource)
* New strategy items (without learning resource)

[[Image:IPL measures.jpg]]

Motivation and affect:
* Behavioral measure: % of students who kept working during breaks
* Self reports

=== Hypothesis ===

One hypothesis argues that generative reasoning (in the form of symbolic invention) is necessary to improve encoding of subsequent instruction. First, generative reasoning facilitates a process in which students express their prior ideas, identify their shortcomings, and refine their mental models, thus enabling conceptual change (Smith, diSessa, & Roschelle, 1994). For example, the self-explanation literature shows that asking students to explain their errors facilitates conceptual shift (c.f., Siegler, 2002).
By attempting to invent and understand how different symbolic procedures succeed (or fail) to capture the differences between the contrasting cases, students also acquire a more cohesive and integrated understanding of the deep features of the domain. The importance of the symbolic nature of the process was demonstrated by Schwartz, Martin, and Pfaffman (2005), who asked students to reason verbally or mathematically about the balance beam problem. All students noticed the deep features of the balance beam domain - distance and weight. However, only students who reasoned mathematically were able to reconcile the two dimensions to a single representation. Interestingly, students’ thinking evolved even though their solutions were not complete, similar to the IPL effect.
Lastly, the generative reasoning process may help students understand the function of the different components of the procedure (for example, dividing by N controls for sample size). Thus, students may encode the subsequent instruction by function and not merely by procedure. Functional mental models were previously shown to lead to better adaptation of knowledge (Kieras & Bovair, 1984). Hatano and Inagaki (1986) describe a similar process in which developing mental models of how procedures interact with empirical knowledge helps students acquire conceptual understanding of the domain.
An alternative hypothesis argues that comparative reasoning is sufficient to achieve the learning benefits of IPL. According to this hypothesis, the benefits of invention stem from noticing and encoding the deep features of the domain. The comparative reasoning activity achieves that benefit by asking students to compare contrasting cases that differ with respect to their deep features. (Bransford & Schwartz, 2001). This qualitative analysis helps students set requirements for a valid model and thus acquire a better understanding (even if implicit) of the target concepts. Furthermore, according to this hypothesis, not only does the symbolic invention not contribute to future learning, it may waste students’ time (and thus reduce efficiency) or impose excessive cognitive load (Kirschner, Sweller & Clark, 2006).
A second research question addressed by our current study examines the effect of IPL on the flexibility of students’ knowledge. We follow a distinction made by McDaniel and Schlager (1990) between transfer problems that require the application of a learned strategy (conventional transfer problems) and transfer problems that require the generation of a new strategy. McDaniel and Schlager found that while discovery tasks improve students’ performance on the latter, they have no effect on conventional transfer problems. Schwartz and Martin (2004) add a twist to these results. They found that IPL improves students’ ability to solve new-strategy problems as long as they are provided with instruction on how to do so. To further investigate the effect of IPL on knowledge flexibility, we evaluate students’ ability to independently solve new-strategy problems and encode new-strategy instructions. Our hypothesis, as supported by McDaniel and Schlager (1990), is that students who are engaged in IPL will acquire more flexible knowledge and thus will demonstrate better performance on new-strategy items. At the same time they will not show better ability to use existing strategies in novel contexts (conventional transfer items). Furthermore, following the findings of Schwartz and Martin (2004), we hypothesize that the effect of IPL will be mainly on encoding new-strategy instructions.

==== Instructional Principles ====
See [[IPL Instructional Principles]]

=== Findings ===

Domain knowledge:

[[Image:test results.jpg]]

* IPL students in advanced classes were more capable of solving new strategy items without learning resource. In fact, in the absence of a learning resource, direct instruction students performed at floor, while IPL students performed as well as with the source.
* This effect holds when controlling for simple domain knowledge (performance on normal items in the same test).
* This was found in multiple new-strategy items. However, all results were found on a single topic (central tendency and graphing). The single test item on the topic of variability failed to capture difference between the groups.

Motivation:
* IPL students reported to have benefited more (F=3.3, p<.07)
* There was a significant interaction between condition and test anxiety. Text anxiety was assessed using the MSLQ (Pintrich 1999) before the study began. Students who reported to have lower test anxiety also reported to have benefited more from IPL instruction compared to high-anxiety students in the no desin condition.
* IPL students stayed more often in class to work during breaks (IPL: 16% No Design: 3%).
* Furthermore, they did so during invention activities and not show-and-practice activities, suggesting that it is the activities that are motivating.

[[Image:motivational results.jpg]]

=== Explanation ===
Regarding our first research question, we found that generative reasoning (on top of comparative reasoning) had a positive effect on students’ ability to solve new-strategy problems with no learning resource in the advanced classes. At the same time, as hypothesized, it had a marginal effect on normal or conventional transfer items. These results are interesting especially since Full IPL students had approximately half the time for instruction and practice compared with their No Design counterparts.
Regarding the second research question, which dealt with students’ knowledge flexibility, we found that in the advanced classes, students who designed novel methods during IPL were more capable of solving problems that require the use of novel strategies. This finding echoes the effect found by McDaniel and Schlager (1990). Interestingly, the effect of IPL on new-strategy items with no resources holds even when controlling for performance on normal items on the same test. Thus, this effect can probably not be attributed to more domain knowledge. Instead, it is likely the outcome of a different encoding of domain knowledge, in a manner that is not reflected in normal or transfer items.
On further scrutiny, students in both conditions did equally well on all tasks for which they received some form of instruction - whether in class (on normal and conventional transfer items) or embedded in the test (on new-strategy items with embedded learning resources). Regarding the latter, it seems that Full IPL students did not need the additional instruction whereas No Design students did not manage to solve the new-strategy problems without it. The performance of Full IPL students on new-strategy items remained virtually the same even in the absence of embedded instruction. This finding is at odds with earlier findings by Schwartz and Taylor (2004) who found that IPL improves students’ ability to encode future instruction but not solve novel problems without additional instruction. One explanation for the discrepancy between the studies is that the control group in Schwartz and Taylor (2004) did not engage in comparative reasoning. Therefore, it may be that the comparative reasoning stage helped students in our study to encode the novel instruction.
An alternative explanation examines these results in terms of ‘distance’ from original classroom instruction. It may be that the embedded instruction on the first topic in our study was close to the classroom material, and thus simple enough for all students to encode. In contrast, the embedded learning resource in the study described by Schwartz and Martin (2004) was sufficiently far from the classroom instruction. Therefore, only IPL students, who had acquired more flexible knowledge, could learn from it and apply the acquired knowledge successfully. This explanation further suggests that in the absence of additional instruction, only Full IPL students in our study could make the leap and answer the target new-strategy items.
While this argument explains performance on new-strategy items (with or without instruction) in terms of distance from classroom instruction, it does not explain what factors determine this distance. What makes some items ‘closer’ than others? What prepared Full IPL students for improved performance on some items but not on others?
Students may grapple with many challenges during the invention phase, many of which do not receive attention during classroom instruction. Students who invent are exposed to various challenges by virtue of attempting to invent general valid methods. We hypothesize that students use knowledge acquired during these experiences when later integrating new-strategy tasks into their existing body of knowledge. For example, the post-tests in this study included three new-strategy items, requiring the following new strategies: (1) comparing multiple datasets in a single representation; (2) representing data in unconventional intervals; and (3) finding the ratio between variability and average in order to account for differences in magnitude. These topics were not covered during classroom instruction. However, when we analyzed students’ inventions, we noticed that many inventions included features that could prepare students to expand the instructed knowledge and invent the first two strategies (see Figure 3). Subsequently, Full IPL students demonstrated better performance on the relevant new-strategy items. At the same time, no student attempted during invention to compare datasets with different magnitudes. Correspondingly, Full IPL students did not exhibit better performance on this new-strategy item.

=== Connections===

* Tim Nokes's study

=== The Invention Lab ===
In addition to paper and pencil studies, we have created the Invention Lab.

The Invention Lab is an intelligent tutoring system for IPL. To give intelligent feedback, it uses two models:
* A meta-cognitive model of the invention process
* A cognitive model of the main concepts in the domain

[[Image:iLab.jpg]]

=== Annotated bibliography ===

[[Category:Empirical Study]]
[[Category:Protected]]

File:Motivational results.jpg

2009-05-21T03:54:10Z

Idoroll:

File:Test results.jpg

2009-05-21T03:53:40Z

Idoroll:

Roll IPL

2009-05-21T03:53:20Z

Idoroll: /* Findings */

== Invention as preparation for learning ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

Can invention activities prepare students to better learn from subsequent instruction, compared with instruction-and-practice only?

=== Meta-data ===

PI's: Ido Roll, Vincent Aleven, Dan Schwartz, Ken Koedinger

Other Contributers: David Klahr
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 4/2007 || 4/2007 || North Hills || 20 || 40 || No, paper-and-pencil only
|-
| '''2''' || 9/2007 || 12/2007 || Community Day || 4 || 48 || No, paper-and-pencil only
|-
| '''3''' || 4/2008 || 5/2008 || Steel Valley || 125 || 900 || No, paper-and-pencil only
|-
| '''4''' || 4/2009 || 5/2009 || Steel Valley || 140 || 560 || Some of it in DataShop, the rest is getting there
|-
|}

=== Abstract ===

The [[assistance dilemma]] asks what form of assistance is most appropriate for different stages of learning. While direct instruction and practice have been shown to be efficient for novices, students often acquire shallow knowledge components and lack robust understanding. Some evidence suggests that invention using contrasting cases, prior to instruction and practice, can accelerate future learning, compared with instruction and practice alone (Schwartz & Martin, 2004).
The invention process as described in Schwartz & Martin (2004) includes the following stages: Design (of a mathematical model to solve a class of problems); calculation (of the solution based on the model); evaluation (of its correctness); and debugging (of the faulty model). Notably, most students fail to invent mathematically valid models, so the goal is not for students to discover the correct solution. At the same time, students do make models that capture deep features of the class of problems, which prepares them to learn and understand the significance of expert solutions for handling such situations. Following the invention, students receive instruction on the expert solution (that is, formulas) and practice it. This procedure is based on the hypothesis that students’ own inventions, together with subsequent instruction, are sources for coordinative learning. By attempting to create a model that correctly distinguishes the “contrasting cases” (carefully selected instances within a class of problems) students notice (and to some degree invent) the problem features that an adequate model must take into account, and they attend to them during subsequent instruction. However, alternative explanations for the effectiveness of the IPL process are possible, with different instructional implications. A “debugging hypothesis” suggests that evaluation and debugging of pre-designed models are sufficient to promote future learning by directing students’ attention to the short-comings of the designed models, and thus to the deep features of the domain. Alternatively, an “unfinished goals” hypothesis suggests that the effect is caused by students reaching impasses during invention. According to this hypothesis, calculation-evaluation are sufficient for preparing for future learning.
We propose to investigate this in a series of ablation studies with the goal of better defining the invention process and identifying the cognitive processes involved. This includes a combination of in-vivo and lab studies within the Algebra LearnLab, contributing to the Coordinative Learning theoretical framework. Following the ablation studies we plan to implement the procedure in a Cognitive Tutor, which will be evaluated in a lab study. This will allow us to better operationalize the process, do a micro-genetic analysis of it, and identify productive patterns of learning trajectories using log mining.

=== Background and Significance ===

One of the main challenges of education is to help students reach meaningful and robust learning. The assistance dilemma raises the question of what form (and ‘amount’) of assistance are most effective with different learners in different stages of the learning process (Koedinger & Aleven, in press). Instruction followed by practice is known to be very efficient for teaching novices (e.g., Koedinger, Anderson, Hadley & Mark, 1997); yet, students often acquire shallow procedural skills, and fail to acquire conceptual understanding (Aleven & Koedinger, 2002). This can be attributed, at least in part, to students using superficial features and not encoding the deep features of the domain (Chi, Feltovich & Glaser, 1981).
One approach to getting students to attend and encode the deep features is to add an invention phase prior to instruction. Invention as preparation for leaning (IPL) was shown to help students better cope with novel situations that require learning (Schwartz & Martin, 2004; Sears, 2006). In this process students are presented with a dilemma in the form of contrasting cases, and attempt to invent a mathematical model to resolve this dilemma. For example, Figure 1 shows four possible pitching machines. Students are asked to invent a method that will allow them to pick the most reliable machine. The concept of contrasting cases comes from the perceptual learning literature, since these cases, when appropriately designed, emphasize differences in the deep structure of the examples (Gibson & Gibson, 1955). The invention process includes designing a model, applying it to the given set of contrasting cases, evaluating the result, and debugging the model. This iterative process is very similar to the debugging process as described by Klahr and Carver (1988; Figure 2). Unlike other inquiry-based manipulations (cf. Lehrer et al., 2001; de Jong & van Joolingen, 1998), the goal of the IPL process is not for students to discover the correct model, but to prepare them for subsequent instruction. During the instruction students share their models, critic their peers’ models, and learn the expert solutions (A similar classroom critic process was shown to be effective by White & Frederiksen, 1998). Preparation for learning from the instruction is evaluated using accelerated future learning assessment. The accelerated future learning assessment includes an embedded instruction in the test in the form of solved example. Schwartz (2004) found that only students who invented prior to the test were able to take advantage of that instruction in order to solve novel problem, while students who practiced a given visual method prior to the test did not take advantage of the embedded learning resource and thus could not solve the target problem. This shows that the IPL process has a positive effect on students’ ability to independently learn from the solved examples. In the case of the contrasting cases given in Figure 1, subsequent instruction will introduce the students with the notion (and formulas) of variance.
While the invention group was superior to instruction-and-practice group on accelerated future learning measure, there was no direct comparison of normal or transfer measures between invention and instruction-and-practice conditions (though invention students showed pre-to-post gains, and were shown to outperformed college students). Also, it is not yet clear how robust this pedagogy is and what its key features are.

Example for contrasting cases (topic - variability)

[[Image:contrasting cases.jpg]]

The overall IPL process:

[[Image:IPL process.jpg]]

=== Glossary ===
See [[:Category:IPL|IPL Glossary]]

=== Research questions ===

# What is the overall effect of [[Invention task]]s on students domain knowledge, sense-making skills, and motivation, compared with [[direct instruction]]?
# What elements of invention contribute to that effect? What cognitive processes do they drive? In what ways does knowledge acquired following invention differ from knowledge acquired in direct instruction alone?
# Can the IPL process be scaled-up using technology?

In addition, the project makes the following contributions:
# It compares different measures of [[robust learning]], in order to understand what aspect of knowledge can be assessed using what type of measure.

=== Independent Variables ===

Different studies manipulated different stages of the [[Invention task]]:
- Observation (a.k.a. comparative reasoning): comparing contrasting cases that vary along deep features, with regard to target concepts
- Generative reasoning: designing novel mathematical procedures to compare the contrasting cases with regard to the target concept
- Evaluation: Evaluation of the models

=== Dependent variables ===

Domain knowledge (in increasing 'distance' from instruction):
* Normal measures
* Transfer measures
* New strategy items (with learning resource)
* New strategy items (without learning resource)

[[Image:IPL measures.jpg]]

Motivation and affect:
* Behavioral measure: % of students who kept working during breaks
* Self reports

=== Hypothesis ===

One hypothesis argues that generative reasoning (in the form of symbolic invention) is necessary to improve encoding of subsequent instruction. First, generative reasoning facilitates a process in which students express their prior ideas, identify their shortcomings, and refine their mental models, thus enabling conceptual change (Smith, diSessa, & Roschelle, 1994). For example, the self-explanation literature shows that asking students to explain their errors facilitates conceptual shift (c.f., Siegler, 2002).
By attempting to invent and understand how different symbolic procedures succeed (or fail) to capture the differences between the contrasting cases, students also acquire a more cohesive and integrated understanding of the deep features of the domain. The importance of the symbolic nature of the process was demonstrated by Schwartz, Martin, and Pfaffman (2005), who asked students to reason verbally or mathematically about the balance beam problem. All students noticed the deep features of the balance beam domain - distance and weight. However, only students who reasoned mathematically were able to reconcile the two dimensions to a single representation. Interestingly, students’ thinking evolved even though their solutions were not complete, similar to the IPL effect.
Lastly, the generative reasoning process may help students understand the function of the different components of the procedure (for example, dividing by N controls for sample size). Thus, students may encode the subsequent instruction by function and not merely by procedure. Functional mental models were previously shown to lead to better adaptation of knowledge (Kieras & Bovair, 1984). Hatano and Inagaki (1986) describe a similar process in which developing mental models of how procedures interact with empirical knowledge helps students acquire conceptual understanding of the domain.
An alternative hypothesis argues that comparative reasoning is sufficient to achieve the learning benefits of IPL. According to this hypothesis, the benefits of invention stem from noticing and encoding the deep features of the domain. The comparative reasoning activity achieves that benefit by asking students to compare contrasting cases that differ with respect to their deep features. (Bransford & Schwartz, 2001). This qualitative analysis helps students set requirements for a valid model and thus acquire a better understanding (even if implicit) of the target concepts. Furthermore, according to this hypothesis, not only does the symbolic invention not contribute to future learning, it may waste students’ time (and thus reduce efficiency) or impose excessive cognitive load (Kirschner, Sweller & Clark, 2006).
A second research question addressed by our current study examines the effect of IPL on the flexibility of students’ knowledge. We follow a distinction made by McDaniel and Schlager (1990) between transfer problems that require the application of a learned strategy (conventional transfer problems) and transfer problems that require the generation of a new strategy. McDaniel and Schlager found that while discovery tasks improve students’ performance on the latter, they have no effect on conventional transfer problems. Schwartz and Martin (2004) add a twist to these results. They found that IPL improves students’ ability to solve new-strategy problems as long as they are provided with instruction on how to do so. To further investigate the effect of IPL on knowledge flexibility, we evaluate students’ ability to independently solve new-strategy problems and encode new-strategy instructions. Our hypothesis, as supported by McDaniel and Schlager (1990), is that students who are engaged in IPL will acquire more flexible knowledge and thus will demonstrate better performance on new-strategy items. At the same time they will not show better ability to use existing strategies in novel contexts (conventional transfer items). Furthermore, following the findings of Schwartz and Martin (2004), we hypothesize that the effect of IPL will be mainly on encoding new-strategy instructions.

==== Instructional Principles ====
See [[IPL Instructional Principles]]

=== Findings ===

Domain knowledge:

[[Image:test results.jpg]]

* IPL students in advanced classes were more capable of solving new strategy items without learning resource. In fact, in the absence of a learning resource, direct instruction students performed at floor, while IPL students performed as well as with the source.
* This effect holds when controlling for simple domain knowledge (performance on normal items in the same test).
* This was found in multiple new-strategy items. However, all results were found on a single topic (central tendency and graphing). The single test item on the topic of variability failed to capture difference between the groups.

Motivation:
* IPL students reported to have benefited more (F=3.3, p<.07)
* There was a significant interaction between condition and test anxiety. Text anxiety was assessed using the MSLQ (Pintrich 1999) before the study began. Students who reported to have lower test anxiety also reported to have benefited more from IPL instruction compared to high-anxiety students in the no desin condition.
* IPL students stayed more often in class to work during breaks (IPL: 16% No Design: 3%).
* Furthermore, they did so during invention activities and not show-and-practice activities, suggesting that it is the activities that are motivating.

[[Image:motivational results.jpg]]

=== Explanation ===
Regarding our first research question, we found that generative reasoning (on top of comparative reasoning) had a positive effect on students’ ability to solve new-strategy problems with no learning resource in the advanced classes. At the same time, as hypothesized, it had a marginal effect on normal or conventional transfer items. These results are interesting especially since Full IPL students had approximately half the time for instruction and practice compared with their No Design counterparts.
Regarding the second research question, which dealt with students’ knowledge flexibility, we found that in the advanced classes, students who designed novel methods during IPL were more capable of solving problems that require the use of novel strategies. This finding echoes the effect found by McDaniel and Schlager (1990). Interestingly, the effect of IPL on new-strategy items with no resources holds even when controlling for performance on normal items on the same test. Thus, this effect can probably not be attributed to more domain knowledge. Instead, it is likely the outcome of a different encoding of domain knowledge, in a manner that is not reflected in normal or transfer items.
On further scrutiny, students in both conditions did equally well on all tasks for which they received some form of instruction - whether in class (on normal and conventional transfer items) or embedded in the test (on new-strategy items with embedded learning resources). Regarding the latter, it seems that Full IPL students did not need the additional instruction whereas No Design students did not manage to solve the new-strategy problems without it. The performance of Full IPL students on new-strategy items remained virtually the same even in the absence of embedded instruction. This finding is at odds with earlier findings by Schwartz and Taylor (2004) who found that IPL improves students’ ability to encode future instruction but not solve novel problems without additional instruction. One explanation for the discrepancy between the studies is that the control group in Schwartz and Taylor (2004) did not engage in comparative reasoning. Therefore, it may be that the comparative reasoning stage helped students in our study to encode the novel instruction.
An alternative explanation examines these results in terms of ‘distance’ from original classroom instruction. It may be that the embedded instruction on the first topic in our study was close to the classroom material, and thus simple enough for all students to encode. In contrast, the embedded learning resource in the study described by Schwartz and Martin (2004) was sufficiently far from the classroom instruction. Therefore, only IPL students, who had acquired more flexible knowledge, could learn from it and apply the acquired knowledge successfully. This explanation further suggests that in the absence of additional instruction, only Full IPL students in our study could make the leap and answer the target new-strategy items.
While this argument explains performance on new-strategy items (with or without instruction) in terms of distance from classroom instruction, it does not explain what factors determine this distance. What makes some items ‘closer’ than others? What prepared Full IPL students for improved performance on some items but not on others?
Students may grapple with many challenges during the invention phase, many of which do not receive attention during classroom instruction. Students who invent are exposed to various challenges by virtue of attempting to invent general valid methods. We hypothesize that students use knowledge acquired during these experiences when later integrating new-strategy tasks into their existing body of knowledge. For example, the post-tests in this study included three new-strategy items, requiring the following new strategies: (1) comparing multiple datasets in a single representation; (2) representing data in unconventional intervals; and (3) finding the ratio between variability and average in order to account for differences in magnitude. These topics were not covered during classroom instruction. However, when we analyzed students’ inventions, we noticed that many inventions included features that could prepare students to expand the instructed knowledge and invent the first two strategies (see Figure 3). Subsequently, Full IPL students demonstrated better performance on the relevant new-strategy items. At the same time, no student attempted during invention to compare datasets with different magnitudes. Correspondingly, Full IPL students did not exhibit better performance on this new-strategy item.

=== Connections===

* Tim Nokes's study

=== Further Information ===

=== Annotated bibliography ===

[[Category:Empirical Study]]
[[Category:Protected]]

File:IPL measures.jpg

2009-05-21T03:52:49Z

Idoroll:

Roll IPL

2009-05-21T03:52:12Z

Idoroll: /* Background and Significance */

== Invention as preparation for learning ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

Can invention activities prepare students to better learn from subsequent instruction, compared with instruction-and-practice only?

=== Meta-data ===

PI's: Ido Roll, Vincent Aleven, Dan Schwartz, Ken Koedinger

Other Contributers: David Klahr
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 4/2007 || 4/2007 || North Hills || 20 || 40 || No, paper-and-pencil only
|-
| '''2''' || 9/2007 || 12/2007 || Community Day || 4 || 48 || No, paper-and-pencil only
|-
| '''3''' || 4/2008 || 5/2008 || Steel Valley || 125 || 900 || No, paper-and-pencil only
|-
| '''4''' || 4/2009 || 5/2009 || Steel Valley || 140 || 560 || Some of it in DataShop, the rest is getting there
|-
|}

=== Abstract ===

The [[assistance dilemma]] asks what form of assistance is most appropriate for different stages of learning. While direct instruction and practice have been shown to be efficient for novices, students often acquire shallow knowledge components and lack robust understanding. Some evidence suggests that invention using contrasting cases, prior to instruction and practice, can accelerate future learning, compared with instruction and practice alone (Schwartz & Martin, 2004).
The invention process as described in Schwartz & Martin (2004) includes the following stages: Design (of a mathematical model to solve a class of problems); calculation (of the solution based on the model); evaluation (of its correctness); and debugging (of the faulty model). Notably, most students fail to invent mathematically valid models, so the goal is not for students to discover the correct solution. At the same time, students do make models that capture deep features of the class of problems, which prepares them to learn and understand the significance of expert solutions for handling such situations. Following the invention, students receive instruction on the expert solution (that is, formulas) and practice it. This procedure is based on the hypothesis that students’ own inventions, together with subsequent instruction, are sources for coordinative learning. By attempting to create a model that correctly distinguishes the “contrasting cases” (carefully selected instances within a class of problems) students notice (and to some degree invent) the problem features that an adequate model must take into account, and they attend to them during subsequent instruction. However, alternative explanations for the effectiveness of the IPL process are possible, with different instructional implications. A “debugging hypothesis” suggests that evaluation and debugging of pre-designed models are sufficient to promote future learning by directing students’ attention to the short-comings of the designed models, and thus to the deep features of the domain. Alternatively, an “unfinished goals” hypothesis suggests that the effect is caused by students reaching impasses during invention. According to this hypothesis, calculation-evaluation are sufficient for preparing for future learning.
We propose to investigate this in a series of ablation studies with the goal of better defining the invention process and identifying the cognitive processes involved. This includes a combination of in-vivo and lab studies within the Algebra LearnLab, contributing to the Coordinative Learning theoretical framework. Following the ablation studies we plan to implement the procedure in a Cognitive Tutor, which will be evaluated in a lab study. This will allow us to better operationalize the process, do a micro-genetic analysis of it, and identify productive patterns of learning trajectories using log mining.

=== Background and Significance ===

One of the main challenges of education is to help students reach meaningful and robust learning. The assistance dilemma raises the question of what form (and ‘amount’) of assistance are most effective with different learners in different stages of the learning process (Koedinger & Aleven, in press). Instruction followed by practice is known to be very efficient for teaching novices (e.g., Koedinger, Anderson, Hadley & Mark, 1997); yet, students often acquire shallow procedural skills, and fail to acquire conceptual understanding (Aleven & Koedinger, 2002). This can be attributed, at least in part, to students using superficial features and not encoding the deep features of the domain (Chi, Feltovich & Glaser, 1981).
One approach to getting students to attend and encode the deep features is to add an invention phase prior to instruction. Invention as preparation for leaning (IPL) was shown to help students better cope with novel situations that require learning (Schwartz & Martin, 2004; Sears, 2006). In this process students are presented with a dilemma in the form of contrasting cases, and attempt to invent a mathematical model to resolve this dilemma. For example, Figure 1 shows four possible pitching machines. Students are asked to invent a method that will allow them to pick the most reliable machine. The concept of contrasting cases comes from the perceptual learning literature, since these cases, when appropriately designed, emphasize differences in the deep structure of the examples (Gibson & Gibson, 1955). The invention process includes designing a model, applying it to the given set of contrasting cases, evaluating the result, and debugging the model. This iterative process is very similar to the debugging process as described by Klahr and Carver (1988; Figure 2). Unlike other inquiry-based manipulations (cf. Lehrer et al., 2001; de Jong & van Joolingen, 1998), the goal of the IPL process is not for students to discover the correct model, but to prepare them for subsequent instruction. During the instruction students share their models, critic their peers’ models, and learn the expert solutions (A similar classroom critic process was shown to be effective by White & Frederiksen, 1998). Preparation for learning from the instruction is evaluated using accelerated future learning assessment. The accelerated future learning assessment includes an embedded instruction in the test in the form of solved example. Schwartz (2004) found that only students who invented prior to the test were able to take advantage of that instruction in order to solve novel problem, while students who practiced a given visual method prior to the test did not take advantage of the embedded learning resource and thus could not solve the target problem. This shows that the IPL process has a positive effect on students’ ability to independently learn from the solved examples. In the case of the contrasting cases given in Figure 1, subsequent instruction will introduce the students with the notion (and formulas) of variance.
While the invention group was superior to instruction-and-practice group on accelerated future learning measure, there was no direct comparison of normal or transfer measures between invention and instruction-and-practice conditions (though invention students showed pre-to-post gains, and were shown to outperformed college students). Also, it is not yet clear how robust this pedagogy is and what its key features are.

Example for contrasting cases (topic - variability)

[[Image:contrasting cases.jpg]]

The overall IPL process:

[[Image:IPL process.jpg]]

=== Glossary ===
See [[:Category:IPL|IPL Glossary]]

=== Research questions ===

# What is the overall effect of [[Invention task]]s on students domain knowledge, sense-making skills, and motivation, compared with [[direct instruction]]?
# What elements of invention contribute to that effect? What cognitive processes do they drive? In what ways does knowledge acquired following invention differ from knowledge acquired in direct instruction alone?
# Can the IPL process be scaled-up using technology?

In addition, the project makes the following contributions:
# It compares different measures of [[robust learning]], in order to understand what aspect of knowledge can be assessed using what type of measure.

=== Independent Variables ===

Different studies manipulated different stages of the [[Invention task]]:
- Observation (a.k.a. comparative reasoning): comparing contrasting cases that vary along deep features, with regard to target concepts
- Generative reasoning: designing novel mathematical procedures to compare the contrasting cases with regard to the target concept
- Evaluation: Evaluation of the models

=== Dependent variables ===

Domain knowledge (in increasing 'distance' from instruction):
* Normal measures
* Transfer measures
* New strategy items (with learning resource)
* New strategy items (without learning resource)

[[Image:IPL measures.jpg]]

Motivation and affect:
* Behavioral measure: % of students who kept working during breaks
* Self reports

=== Hypothesis ===

One hypothesis argues that generative reasoning (in the form of symbolic invention) is necessary to improve encoding of subsequent instruction. First, generative reasoning facilitates a process in which students express their prior ideas, identify their shortcomings, and refine their mental models, thus enabling conceptual change (Smith, diSessa, & Roschelle, 1994). For example, the self-explanation literature shows that asking students to explain their errors facilitates conceptual shift (c.f., Siegler, 2002).
By attempting to invent and understand how different symbolic procedures succeed (or fail) to capture the differences between the contrasting cases, students also acquire a more cohesive and integrated understanding of the deep features of the domain. The importance of the symbolic nature of the process was demonstrated by Schwartz, Martin, and Pfaffman (2005), who asked students to reason verbally or mathematically about the balance beam problem. All students noticed the deep features of the balance beam domain - distance and weight. However, only students who reasoned mathematically were able to reconcile the two dimensions to a single representation. Interestingly, students’ thinking evolved even though their solutions were not complete, similar to the IPL effect.
Lastly, the generative reasoning process may help students understand the function of the different components of the procedure (for example, dividing by N controls for sample size). Thus, students may encode the subsequent instruction by function and not merely by procedure. Functional mental models were previously shown to lead to better adaptation of knowledge (Kieras & Bovair, 1984). Hatano and Inagaki (1986) describe a similar process in which developing mental models of how procedures interact with empirical knowledge helps students acquire conceptual understanding of the domain.
An alternative hypothesis argues that comparative reasoning is sufficient to achieve the learning benefits of IPL. According to this hypothesis, the benefits of invention stem from noticing and encoding the deep features of the domain. The comparative reasoning activity achieves that benefit by asking students to compare contrasting cases that differ with respect to their deep features. (Bransford & Schwartz, 2001). This qualitative analysis helps students set requirements for a valid model and thus acquire a better understanding (even if implicit) of the target concepts. Furthermore, according to this hypothesis, not only does the symbolic invention not contribute to future learning, it may waste students’ time (and thus reduce efficiency) or impose excessive cognitive load (Kirschner, Sweller & Clark, 2006).
A second research question addressed by our current study examines the effect of IPL on the flexibility of students’ knowledge. We follow a distinction made by McDaniel and Schlager (1990) between transfer problems that require the application of a learned strategy (conventional transfer problems) and transfer problems that require the generation of a new strategy. McDaniel and Schlager found that while discovery tasks improve students’ performance on the latter, they have no effect on conventional transfer problems. Schwartz and Martin (2004) add a twist to these results. They found that IPL improves students’ ability to solve new-strategy problems as long as they are provided with instruction on how to do so. To further investigate the effect of IPL on knowledge flexibility, we evaluate students’ ability to independently solve new-strategy problems and encode new-strategy instructions. Our hypothesis, as supported by McDaniel and Schlager (1990), is that students who are engaged in IPL will acquire more flexible knowledge and thus will demonstrate better performance on new-strategy items. At the same time they will not show better ability to use existing strategies in novel contexts (conventional transfer items). Furthermore, following the findings of Schwartz and Martin (2004), we hypothesize that the effect of IPL will be mainly on encoding new-strategy instructions.

==== Instructional Principles ====
See [[IPL Instructional Principles]]

=== Findings ===

Domain knowledge:
[[Image:test results.jpg]]
* IPL students in advanced classes were more capable of solving new strategy items without learning resource. In fact, in the absence of a learning resource, direct instruction students performed at floor, while IPL students performed as well as with the source.
* This effect holds when controlling for simple domain knowledge (performance on normal items in the same test).
* This was found in multiple new-strategy items. However, all results were found on a single topic (central tendency and graphing). The single test item on the topic of variability failed to capture difference between the groups.

Motivation:
* IPL students reported to have benefited more (F=3.3, p<.07)
* There was a significant interaction between condition and test anxiety. Text anxiety was assessed using the MSLQ (Pintrich 1999) before the study began. Students who reported to have lower test anxiety also reported to have benefited more from IPL instruction compared to high-anxiety students in the no desin condition.
* IPL students stayed more often in class to work during breaks (IPL: 16% No Design: 3%).
* Furthermore, they did so during invention activities and not show-and-practice activities, suggesting that it is the activities that are motivating.
[[Image:motivational results.jpg]]

=== Explanation ===
Regarding our first research question, we found that generative reasoning (on top of comparative reasoning) had a positive effect on students’ ability to solve new-strategy problems with no learning resource in the advanced classes. At the same time, as hypothesized, it had a marginal effect on normal or conventional transfer items. These results are interesting especially since Full IPL students had approximately half the time for instruction and practice compared with their No Design counterparts.
Regarding the second research question, which dealt with students’ knowledge flexibility, we found that in the advanced classes, students who designed novel methods during IPL were more capable of solving problems that require the use of novel strategies. This finding echoes the effect found by McDaniel and Schlager (1990). Interestingly, the effect of IPL on new-strategy items with no resources holds even when controlling for performance on normal items on the same test. Thus, this effect can probably not be attributed to more domain knowledge. Instead, it is likely the outcome of a different encoding of domain knowledge, in a manner that is not reflected in normal or transfer items.
On further scrutiny, students in both conditions did equally well on all tasks for which they received some form of instruction - whether in class (on normal and conventional transfer items) or embedded in the test (on new-strategy items with embedded learning resources). Regarding the latter, it seems that Full IPL students did not need the additional instruction whereas No Design students did not manage to solve the new-strategy problems without it. The performance of Full IPL students on new-strategy items remained virtually the same even in the absence of embedded instruction. This finding is at odds with earlier findings by Schwartz and Taylor (2004) who found that IPL improves students’ ability to encode future instruction but not solve novel problems without additional instruction. One explanation for the discrepancy between the studies is that the control group in Schwartz and Taylor (2004) did not engage in comparative reasoning. Therefore, it may be that the comparative reasoning stage helped students in our study to encode the novel instruction.
An alternative explanation examines these results in terms of ‘distance’ from original classroom instruction. It may be that the embedded instruction on the first topic in our study was close to the classroom material, and thus simple enough for all students to encode. In contrast, the embedded learning resource in the study described by Schwartz and Martin (2004) was sufficiently far from the classroom instruction. Therefore, only IPL students, who had acquired more flexible knowledge, could learn from it and apply the acquired knowledge successfully. This explanation further suggests that in the absence of additional instruction, only Full IPL students in our study could make the leap and answer the target new-strategy items.
While this argument explains performance on new-strategy items (with or without instruction) in terms of distance from classroom instruction, it does not explain what factors determine this distance. What makes some items ‘closer’ than others? What prepared Full IPL students for improved performance on some items but not on others?
Students may grapple with many challenges during the invention phase, many of which do not receive attention during classroom instruction. Students who invent are exposed to various challenges by virtue of attempting to invent general valid methods. We hypothesize that students use knowledge acquired during these experiences when later integrating new-strategy tasks into their existing body of knowledge. For example, the post-tests in this study included three new-strategy items, requiring the following new strategies: (1) comparing multiple datasets in a single representation; (2) representing data in unconventional intervals; and (3) finding the ratio between variability and average in order to account for differences in magnitude. These topics were not covered during classroom instruction. However, when we analyzed students’ inventions, we noticed that many inventions included features that could prepare students to expand the instructed knowledge and invent the first two strategies (see Figure 3). Subsequently, Full IPL students demonstrated better performance on the relevant new-strategy items. At the same time, no student attempted during invention to compare datasets with different magnitudes. Correspondingly, Full IPL students did not exhibit better performance on this new-strategy item.

=== Connections===

* Tim Nokes's study

=== Further Information ===

=== Annotated bibliography ===

[[Category:Empirical Study]]
[[Category:Protected]]

Roll IPL

2009-05-21T03:51:59Z

Idoroll: /* Background and Significance */

== Invention as preparation for learning ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

Can invention activities prepare students to better learn from subsequent instruction, compared with instruction-and-practice only?

=== Meta-data ===

PI's: Ido Roll, Vincent Aleven, Dan Schwartz, Ken Koedinger

Other Contributers: David Klahr
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 4/2007 || 4/2007 || North Hills || 20 || 40 || No, paper-and-pencil only
|-
| '''2''' || 9/2007 || 12/2007 || Community Day || 4 || 48 || No, paper-and-pencil only
|-
| '''3''' || 4/2008 || 5/2008 || Steel Valley || 125 || 900 || No, paper-and-pencil only
|-
| '''4''' || 4/2009 || 5/2009 || Steel Valley || 140 || 560 || Some of it in DataShop, the rest is getting there
|-
|}

=== Abstract ===

The [[assistance dilemma]] asks what form of assistance is most appropriate for different stages of learning. While direct instruction and practice have been shown to be efficient for novices, students often acquire shallow knowledge components and lack robust understanding. Some evidence suggests that invention using contrasting cases, prior to instruction and practice, can accelerate future learning, compared with instruction and practice alone (Schwartz & Martin, 2004).
The invention process as described in Schwartz & Martin (2004) includes the following stages: Design (of a mathematical model to solve a class of problems); calculation (of the solution based on the model); evaluation (of its correctness); and debugging (of the faulty model). Notably, most students fail to invent mathematically valid models, so the goal is not for students to discover the correct solution. At the same time, students do make models that capture deep features of the class of problems, which prepares them to learn and understand the significance of expert solutions for handling such situations. Following the invention, students receive instruction on the expert solution (that is, formulas) and practice it. This procedure is based on the hypothesis that students’ own inventions, together with subsequent instruction, are sources for coordinative learning. By attempting to create a model that correctly distinguishes the “contrasting cases” (carefully selected instances within a class of problems) students notice (and to some degree invent) the problem features that an adequate model must take into account, and they attend to them during subsequent instruction. However, alternative explanations for the effectiveness of the IPL process are possible, with different instructional implications. A “debugging hypothesis” suggests that evaluation and debugging of pre-designed models are sufficient to promote future learning by directing students’ attention to the short-comings of the designed models, and thus to the deep features of the domain. Alternatively, an “unfinished goals” hypothesis suggests that the effect is caused by students reaching impasses during invention. According to this hypothesis, calculation-evaluation are sufficient for preparing for future learning.
We propose to investigate this in a series of ablation studies with the goal of better defining the invention process and identifying the cognitive processes involved. This includes a combination of in-vivo and lab studies within the Algebra LearnLab, contributing to the Coordinative Learning theoretical framework. Following the ablation studies we plan to implement the procedure in a Cognitive Tutor, which will be evaluated in a lab study. This will allow us to better operationalize the process, do a micro-genetic analysis of it, and identify productive patterns of learning trajectories using log mining.

=== Background and Significance ===

One of the main challenges of education is to help students reach meaningful and robust learning. The assistance dilemma raises the question of what form (and ‘amount’) of assistance are most effective with different learners in different stages of the learning process (Koedinger & Aleven, in press). Instruction followed by practice is known to be very efficient for teaching novices (e.g., Koedinger, Anderson, Hadley & Mark, 1997); yet, students often acquire shallow procedural skills, and fail to acquire conceptual understanding (Aleven & Koedinger, 2002). This can be attributed, at least in part, to students using superficial features and not encoding the deep features of the domain (Chi, Feltovich & Glaser, 1981).
One approach to getting students to attend and encode the deep features is to add an invention phase prior to instruction. Invention as preparation for leaning (IPL) was shown to help students better cope with novel situations that require learning (Schwartz & Martin, 2004; Sears, 2006). In this process students are presented with a dilemma in the form of contrasting cases, and attempt to invent a mathematical model to resolve this dilemma. For example, Figure 1 shows four possible pitching machines. Students are asked to invent a method that will allow them to pick the most reliable machine. The concept of contrasting cases comes from the perceptual learning literature, since these cases, when appropriately designed, emphasize differences in the deep structure of the examples (Gibson & Gibson, 1955). The invention process includes designing a model, applying it to the given set of contrasting cases, evaluating the result, and debugging the model. This iterative process is very similar to the debugging process as described by Klahr and Carver (1988; Figure 2). Unlike other inquiry-based manipulations (cf. Lehrer et al., 2001; de Jong & van Joolingen, 1998), the goal of the IPL process is not for students to discover the correct model, but to prepare them for subsequent instruction. During the instruction students share their models, critic their peers’ models, and learn the expert solutions (A similar classroom critic process was shown to be effective by White & Frederiksen, 1998). Preparation for learning from the instruction is evaluated using accelerated future learning assessment. The accelerated future learning assessment includes an embedded instruction in the test in the form of solved example. Schwartz (2004) found that only students who invented prior to the test were able to take advantage of that instruction in order to solve novel problem, while students who practiced a given visual method prior to the test did not take advantage of the embedded learning resource and thus could not solve the target problem. This shows that the IPL process has a positive effect on students’ ability to independently learn from the solved examples. In the case of the contrasting cases given in Figure 1, subsequent instruction will introduce the students with the notion (and formulas) of variance.
While the invention group was superior to instruction-and-practice group on accelerated future learning measure, there was no direct comparison of normal or transfer measures between invention and instruction-and-practice conditions (though invention students showed pre-to-post gains, and were shown to outperformed college students). Also, it is not yet clear how robust this pedagogy is and what its key features are.

Example for contrasting cases (topic - variability)

[[Image:contrasting cases.jpg]]

The overall IPL process:

[[Image:IPL process.jpg]]

=== Glossary ===
See [[:Category:IPL|IPL Glossary]]

=== Research questions ===

# What is the overall effect of [[Invention task]]s on students domain knowledge, sense-making skills, and motivation, compared with [[direct instruction]]?
# What elements of invention contribute to that effect? What cognitive processes do they drive? In what ways does knowledge acquired following invention differ from knowledge acquired in direct instruction alone?
# Can the IPL process be scaled-up using technology?

In addition, the project makes the following contributions:
# It compares different measures of [[robust learning]], in order to understand what aspect of knowledge can be assessed using what type of measure.

=== Independent Variables ===

Different studies manipulated different stages of the [[Invention task]]:
- Observation (a.k.a. comparative reasoning): comparing contrasting cases that vary along deep features, with regard to target concepts
- Generative reasoning: designing novel mathematical procedures to compare the contrasting cases with regard to the target concept
- Evaluation: Evaluation of the models

=== Dependent variables ===

Domain knowledge (in increasing 'distance' from instruction):
* Normal measures
* Transfer measures
* New strategy items (with learning resource)
* New strategy items (without learning resource)

[[Image:IPL measures.jpg]]

Motivation and affect:
* Behavioral measure: % of students who kept working during breaks
* Self reports

=== Hypothesis ===

One hypothesis argues that generative reasoning (in the form of symbolic invention) is necessary to improve encoding of subsequent instruction. First, generative reasoning facilitates a process in which students express their prior ideas, identify their shortcomings, and refine their mental models, thus enabling conceptual change (Smith, diSessa, & Roschelle, 1994). For example, the self-explanation literature shows that asking students to explain their errors facilitates conceptual shift (c.f., Siegler, 2002).
By attempting to invent and understand how different symbolic procedures succeed (or fail) to capture the differences between the contrasting cases, students also acquire a more cohesive and integrated understanding of the deep features of the domain. The importance of the symbolic nature of the process was demonstrated by Schwartz, Martin, and Pfaffman (2005), who asked students to reason verbally or mathematically about the balance beam problem. All students noticed the deep features of the balance beam domain - distance and weight. However, only students who reasoned mathematically were able to reconcile the two dimensions to a single representation. Interestingly, students’ thinking evolved even though their solutions were not complete, similar to the IPL effect.
Lastly, the generative reasoning process may help students understand the function of the different components of the procedure (for example, dividing by N controls for sample size). Thus, students may encode the subsequent instruction by function and not merely by procedure. Functional mental models were previously shown to lead to better adaptation of knowledge (Kieras & Bovair, 1984). Hatano and Inagaki (1986) describe a similar process in which developing mental models of how procedures interact with empirical knowledge helps students acquire conceptual understanding of the domain.
An alternative hypothesis argues that comparative reasoning is sufficient to achieve the learning benefits of IPL. According to this hypothesis, the benefits of invention stem from noticing and encoding the deep features of the domain. The comparative reasoning activity achieves that benefit by asking students to compare contrasting cases that differ with respect to their deep features. (Bransford & Schwartz, 2001). This qualitative analysis helps students set requirements for a valid model and thus acquire a better understanding (even if implicit) of the target concepts. Furthermore, according to this hypothesis, not only does the symbolic invention not contribute to future learning, it may waste students’ time (and thus reduce efficiency) or impose excessive cognitive load (Kirschner, Sweller & Clark, 2006).
A second research question addressed by our current study examines the effect of IPL on the flexibility of students’ knowledge. We follow a distinction made by McDaniel and Schlager (1990) between transfer problems that require the application of a learned strategy (conventional transfer problems) and transfer problems that require the generation of a new strategy. McDaniel and Schlager found that while discovery tasks improve students’ performance on the latter, they have no effect on conventional transfer problems. Schwartz and Martin (2004) add a twist to these results. They found that IPL improves students’ ability to solve new-strategy problems as long as they are provided with instruction on how to do so. To further investigate the effect of IPL on knowledge flexibility, we evaluate students’ ability to independently solve new-strategy problems and encode new-strategy instructions. Our hypothesis, as supported by McDaniel and Schlager (1990), is that students who are engaged in IPL will acquire more flexible knowledge and thus will demonstrate better performance on new-strategy items. At the same time they will not show better ability to use existing strategies in novel contexts (conventional transfer items). Furthermore, following the findings of Schwartz and Martin (2004), we hypothesize that the effect of IPL will be mainly on encoding new-strategy instructions.

==== Instructional Principles ====
See [[IPL Instructional Principles]]

=== Findings ===

Domain knowledge:
[[Image:test results.jpg]]
* IPL students in advanced classes were more capable of solving new strategy items without learning resource. In fact, in the absence of a learning resource, direct instruction students performed at floor, while IPL students performed as well as with the source.
* This effect holds when controlling for simple domain knowledge (performance on normal items in the same test).
* This was found in multiple new-strategy items. However, all results were found on a single topic (central tendency and graphing). The single test item on the topic of variability failed to capture difference between the groups.

Motivation:
* IPL students reported to have benefited more (F=3.3, p<.07)
* There was a significant interaction between condition and test anxiety. Text anxiety was assessed using the MSLQ (Pintrich 1999) before the study began. Students who reported to have lower test anxiety also reported to have benefited more from IPL instruction compared to high-anxiety students in the no desin condition.
* IPL students stayed more often in class to work during breaks (IPL: 16% No Design: 3%).
* Furthermore, they did so during invention activities and not show-and-practice activities, suggesting that it is the activities that are motivating.
[[Image:motivational results.jpg]]

=== Explanation ===
Regarding our first research question, we found that generative reasoning (on top of comparative reasoning) had a positive effect on students’ ability to solve new-strategy problems with no learning resource in the advanced classes. At the same time, as hypothesized, it had a marginal effect on normal or conventional transfer items. These results are interesting especially since Full IPL students had approximately half the time for instruction and practice compared with their No Design counterparts.
Regarding the second research question, which dealt with students’ knowledge flexibility, we found that in the advanced classes, students who designed novel methods during IPL were more capable of solving problems that require the use of novel strategies. This finding echoes the effect found by McDaniel and Schlager (1990). Interestingly, the effect of IPL on new-strategy items with no resources holds even when controlling for performance on normal items on the same test. Thus, this effect can probably not be attributed to more domain knowledge. Instead, it is likely the outcome of a different encoding of domain knowledge, in a manner that is not reflected in normal or transfer items.
On further scrutiny, students in both conditions did equally well on all tasks for which they received some form of instruction - whether in class (on normal and conventional transfer items) or embedded in the test (on new-strategy items with embedded learning resources). Regarding the latter, it seems that Full IPL students did not need the additional instruction whereas No Design students did not manage to solve the new-strategy problems without it. The performance of Full IPL students on new-strategy items remained virtually the same even in the absence of embedded instruction. This finding is at odds with earlier findings by Schwartz and Taylor (2004) who found that IPL improves students’ ability to encode future instruction but not solve novel problems without additional instruction. One explanation for the discrepancy between the studies is that the control group in Schwartz and Taylor (2004) did not engage in comparative reasoning. Therefore, it may be that the comparative reasoning stage helped students in our study to encode the novel instruction.
An alternative explanation examines these results in terms of ‘distance’ from original classroom instruction. It may be that the embedded instruction on the first topic in our study was close to the classroom material, and thus simple enough for all students to encode. In contrast, the embedded learning resource in the study described by Schwartz and Martin (2004) was sufficiently far from the classroom instruction. Therefore, only IPL students, who had acquired more flexible knowledge, could learn from it and apply the acquired knowledge successfully. This explanation further suggests that in the absence of additional instruction, only Full IPL students in our study could make the leap and answer the target new-strategy items.
While this argument explains performance on new-strategy items (with or without instruction) in terms of distance from classroom instruction, it does not explain what factors determine this distance. What makes some items ‘closer’ than others? What prepared Full IPL students for improved performance on some items but not on others?
Students may grapple with many challenges during the invention phase, many of which do not receive attention during classroom instruction. Students who invent are exposed to various challenges by virtue of attempting to invent general valid methods. We hypothesize that students use knowledge acquired during these experiences when later integrating new-strategy tasks into their existing body of knowledge. For example, the post-tests in this study included three new-strategy items, requiring the following new strategies: (1) comparing multiple datasets in a single representation; (2) representing data in unconventional intervals; and (3) finding the ratio between variability and average in order to account for differences in magnitude. These topics were not covered during classroom instruction. However, when we analyzed students’ inventions, we noticed that many inventions included features that could prepare students to expand the instructed knowledge and invent the first two strategies (see Figure 3). Subsequently, Full IPL students demonstrated better performance on the relevant new-strategy items. At the same time, no student attempted during invention to compare datasets with different magnitudes. Correspondingly, Full IPL students did not exhibit better performance on this new-strategy item.

=== Connections===

* Tim Nokes's study

=== Further Information ===

=== Annotated bibliography ===

[[Category:Empirical Study]]
[[Category:Protected]]

File:IPL process.jpg

2009-05-21T03:51:39Z

Idoroll:

File:Contrasting cases.jpg

2009-05-21T03:50:58Z

Idoroll:

Roll IPL

2009-05-21T03:34:14Z

Idoroll:

== Invention as preparation for learning ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

Can invention activities prepare students to better learn from subsequent instruction, compared with instruction-and-practice only?

=== Meta-data ===

PI's: Ido Roll, Vincent Aleven, Dan Schwartz, Ken Koedinger

Other Contributers: David Klahr
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 4/2007 || 4/2007 || North Hills || 20 || 40 || No, paper-and-pencil only
|-
| '''2''' || 9/2007 || 12/2007 || Community Day || 4 || 48 || No, paper-and-pencil only
|-
| '''3''' || 4/2008 || 5/2008 || Steel Valley || 125 || 900 || No, paper-and-pencil only
|-
| '''4''' || 4/2009 || 5/2009 || Steel Valley || 140 || 560 || Some of it in DataShop, the rest is getting there
|-
|}

=== Abstract ===

The [[assistance dilemma]] asks what form of assistance is most appropriate for different stages of learning. While direct instruction and practice have been shown to be efficient for novices, students often acquire shallow knowledge components and lack robust understanding. Some evidence suggests that invention using contrasting cases, prior to instruction and practice, can accelerate future learning, compared with instruction and practice alone (Schwartz & Martin, 2004).
The invention process as described in Schwartz & Martin (2004) includes the following stages: Design (of a mathematical model to solve a class of problems); calculation (of the solution based on the model); evaluation (of its correctness); and debugging (of the faulty model). Notably, most students fail to invent mathematically valid models, so the goal is not for students to discover the correct solution. At the same time, students do make models that capture deep features of the class of problems, which prepares them to learn and understand the significance of expert solutions for handling such situations. Following the invention, students receive instruction on the expert solution (that is, formulas) and practice it. This procedure is based on the hypothesis that students’ own inventions, together with subsequent instruction, are sources for coordinative learning. By attempting to create a model that correctly distinguishes the “contrasting cases” (carefully selected instances within a class of problems) students notice (and to some degree invent) the problem features that an adequate model must take into account, and they attend to them during subsequent instruction. However, alternative explanations for the effectiveness of the IPL process are possible, with different instructional implications. A “debugging hypothesis” suggests that evaluation and debugging of pre-designed models are sufficient to promote future learning by directing students’ attention to the short-comings of the designed models, and thus to the deep features of the domain. Alternatively, an “unfinished goals” hypothesis suggests that the effect is caused by students reaching impasses during invention. According to this hypothesis, calculation-evaluation are sufficient for preparing for future learning.
We propose to investigate this in a series of ablation studies with the goal of better defining the invention process and identifying the cognitive processes involved. This includes a combination of in-vivo and lab studies within the Algebra LearnLab, contributing to the Coordinative Learning theoretical framework. Following the ablation studies we plan to implement the procedure in a Cognitive Tutor, which will be evaluated in a lab study. This will allow us to better operationalize the process, do a micro-genetic analysis of it, and identify productive patterns of learning trajectories using log mining.

=== Background and Significance ===

One of the main challenges of education is to help students reach meaningful and robust learning. The assistance dilemma raises the question of what form (and ‘amount’) of assistance are most effective with different learners in different stages of the learning process (Koedinger & Aleven, in press). Instruction followed by practice is known to be very efficient for teaching novices (e.g., Koedinger, Anderson, Hadley & Mark, 1997); yet, students often acquire shallow procedural skills, and fail to acquire conceptual understanding (Aleven & Koedinger, 2002). This can be attributed, at least in part, to students using superficial features and not encoding the deep features of the domain (Chi, Feltovich & Glaser, 1981).
One approach to getting students to attend and encode the deep features is to add an invention phase prior to instruction. Invention as preparation for leaning (IPL) was shown to help students better cope with novel situations that require learning (Schwartz & Martin, 2004; Sears, 2006). In this process students are presented with a dilemma in the form of contrasting cases, and attempt to invent a mathematical model to resolve this dilemma. For example, Figure 1 shows four possible pitching machines. Students are asked to invent a method that will allow them to pick the most reliable machine. The concept of contrasting cases comes from the perceptual learning literature, since these cases, when appropriately designed, emphasize differences in the deep structure of the examples (Gibson & Gibson, 1955). The invention process includes designing a model, applying it to the given set of contrasting cases, evaluating the result, and debugging the model. This iterative process is very similar to the debugging process as described by Klahr and Carver (1988; Figure 2). Unlike other inquiry-based manipulations (cf. Lehrer et al., 2001; de Jong & van Joolingen, 1998), the goal of the IPL process is not for students to discover the correct model, but to prepare them for subsequent instruction. During the instruction students share their models, critic their peers’ models, and learn the expert solutions (A similar classroom critic process was shown to be effective by White & Frederiksen, 1998). Preparation for learning from the instruction is evaluated using accelerated future learning assessment. The accelerated future learning assessment includes an embedded instruction in the test in the form of solved example. Schwartz (2004) found that only students who invented prior to the test were able to take advantage of that instruction in order to solve novel problem, while students who practiced a given visual method prior to the test did not take advantage of the embedded learning resource and thus could not solve the target problem. This shows that the IPL process has a positive effect on students’ ability to independently learn from the solved examples. In the case of the contrasting cases given in Figure 1, subsequent instruction will introduce the students with the notion (and formulas) of variance.
While the invention group was superior to instruction-and-practice group on accelerated future learning measure, there was no direct comparison of normal or transfer measures between invention and instruction-and-practice conditions (though invention students showed pre-to-post gains, and were shown to outperformed college students). Also, it is not yet clear how robust this pedagogy is and what its key features are.

Example for contrasting cases (topic - variability)
[[Image:contrasting cases.jpg]]

The overall IPL process:
[[Image:IPL process.jpg]]

=== Glossary ===
See [[:Category:IPL|IPL Glossary]]

=== Research questions ===

# What is the overall effect of [[Invention task]]s on students domain knowledge, sense-making skills, and motivation, compared with [[direct instruction]]?
# What elements of invention contribute to that effect? What cognitive processes do they drive? In what ways does knowledge acquired following invention differ from knowledge acquired in direct instruction alone?
# Can the IPL process be scaled-up using technology?

In addition, the project makes the following contributions:
# It compares different measures of [[robust learning]], in order to understand what aspect of knowledge can be assessed using what type of measure.

=== Independent Variables ===

Different studies manipulated different stages of the [[Invention task]]:
- Observation (a.k.a. comparative reasoning): comparing contrasting cases that vary along deep features, with regard to target concepts
- Generative reasoning: designing novel mathematical procedures to compare the contrasting cases with regard to the target concept
- Evaluation: Evaluation of the models

=== Dependent variables ===

Domain knowledge (in increasing 'distance' from instruction):
* Normal measures
* Transfer measures
* New strategy items (with learning resource)
* New strategy items (without learning resource)

[[Image:IPL measures.jpg]]

Motivation and affect:
* Behavioral measure: % of students who kept working during breaks
* Self reports

=== Hypothesis ===

One hypothesis argues that generative reasoning (in the form of symbolic invention) is necessary to improve encoding of subsequent instruction. First, generative reasoning facilitates a process in which students express their prior ideas, identify their shortcomings, and refine their mental models, thus enabling conceptual change (Smith, diSessa, & Roschelle, 1994). For example, the self-explanation literature shows that asking students to explain their errors facilitates conceptual shift (c.f., Siegler, 2002).
By attempting to invent and understand how different symbolic procedures succeed (or fail) to capture the differences between the contrasting cases, students also acquire a more cohesive and integrated understanding of the deep features of the domain. The importance of the symbolic nature of the process was demonstrated by Schwartz, Martin, and Pfaffman (2005), who asked students to reason verbally or mathematically about the balance beam problem. All students noticed the deep features of the balance beam domain - distance and weight. However, only students who reasoned mathematically were able to reconcile the two dimensions to a single representation. Interestingly, students’ thinking evolved even though their solutions were not complete, similar to the IPL effect.
Lastly, the generative reasoning process may help students understand the function of the different components of the procedure (for example, dividing by N controls for sample size). Thus, students may encode the subsequent instruction by function and not merely by procedure. Functional mental models were previously shown to lead to better adaptation of knowledge (Kieras & Bovair, 1984). Hatano and Inagaki (1986) describe a similar process in which developing mental models of how procedures interact with empirical knowledge helps students acquire conceptual understanding of the domain.
An alternative hypothesis argues that comparative reasoning is sufficient to achieve the learning benefits of IPL. According to this hypothesis, the benefits of invention stem from noticing and encoding the deep features of the domain. The comparative reasoning activity achieves that benefit by asking students to compare contrasting cases that differ with respect to their deep features. (Bransford & Schwartz, 2001). This qualitative analysis helps students set requirements for a valid model and thus acquire a better understanding (even if implicit) of the target concepts. Furthermore, according to this hypothesis, not only does the symbolic invention not contribute to future learning, it may waste students’ time (and thus reduce efficiency) or impose excessive cognitive load (Kirschner, Sweller & Clark, 2006).
A second research question addressed by our current study examines the effect of IPL on the flexibility of students’ knowledge. We follow a distinction made by McDaniel and Schlager (1990) between transfer problems that require the application of a learned strategy (conventional transfer problems) and transfer problems that require the generation of a new strategy. McDaniel and Schlager found that while discovery tasks improve students’ performance on the latter, they have no effect on conventional transfer problems. Schwartz and Martin (2004) add a twist to these results. They found that IPL improves students’ ability to solve new-strategy problems as long as they are provided with instruction on how to do so. To further investigate the effect of IPL on knowledge flexibility, we evaluate students’ ability to independently solve new-strategy problems and encode new-strategy instructions. Our hypothesis, as supported by McDaniel and Schlager (1990), is that students who are engaged in IPL will acquire more flexible knowledge and thus will demonstrate better performance on new-strategy items. At the same time they will not show better ability to use existing strategies in novel contexts (conventional transfer items). Furthermore, following the findings of Schwartz and Martin (2004), we hypothesize that the effect of IPL will be mainly on encoding new-strategy instructions.

==== Instructional Principles ====
See [[IPL Instructional Principles]]

=== Findings ===

Domain knowledge:
[[Image:test results.jpg]]
* IPL students in advanced classes were more capable of solving new strategy items without learning resource. In fact, in the absence of a learning resource, direct instruction students performed at floor, while IPL students performed as well as with the source.
* This effect holds when controlling for simple domain knowledge (performance on normal items in the same test).
* This was found in multiple new-strategy items. However, all results were found on a single topic (central tendency and graphing). The single test item on the topic of variability failed to capture difference between the groups.

Motivation:
* IPL students reported to have benefited more (F=3.3, p<.07)
* There was a significant interaction between condition and test anxiety. Text anxiety was assessed using the MSLQ (Pintrich 1999) before the study began. Students who reported to have lower test anxiety also reported to have benefited more from IPL instruction compared to high-anxiety students in the no desin condition.
* IPL students stayed more often in class to work during breaks (IPL: 16% No Design: 3%).
* Furthermore, they did so during invention activities and not show-and-practice activities, suggesting that it is the activities that are motivating.
[[Image:motivational results.jpg]]

=== Explanation ===
Regarding our first research question, we found that generative reasoning (on top of comparative reasoning) had a positive effect on students’ ability to solve new-strategy problems with no learning resource in the advanced classes. At the same time, as hypothesized, it had a marginal effect on normal or conventional transfer items. These results are interesting especially since Full IPL students had approximately half the time for instruction and practice compared with their No Design counterparts.
Regarding the second research question, which dealt with students’ knowledge flexibility, we found that in the advanced classes, students who designed novel methods during IPL were more capable of solving problems that require the use of novel strategies. This finding echoes the effect found by McDaniel and Schlager (1990). Interestingly, the effect of IPL on new-strategy items with no resources holds even when controlling for performance on normal items on the same test. Thus, this effect can probably not be attributed to more domain knowledge. Instead, it is likely the outcome of a different encoding of domain knowledge, in a manner that is not reflected in normal or transfer items.
On further scrutiny, students in both conditions did equally well on all tasks for which they received some form of instruction - whether in class (on normal and conventional transfer items) or embedded in the test (on new-strategy items with embedded learning resources). Regarding the latter, it seems that Full IPL students did not need the additional instruction whereas No Design students did not manage to solve the new-strategy problems without it. The performance of Full IPL students on new-strategy items remained virtually the same even in the absence of embedded instruction. This finding is at odds with earlier findings by Schwartz and Taylor (2004) who found that IPL improves students’ ability to encode future instruction but not solve novel problems without additional instruction. One explanation for the discrepancy between the studies is that the control group in Schwartz and Taylor (2004) did not engage in comparative reasoning. Therefore, it may be that the comparative reasoning stage helped students in our study to encode the novel instruction.
An alternative explanation examines these results in terms of ‘distance’ from original classroom instruction. It may be that the embedded instruction on the first topic in our study was close to the classroom material, and thus simple enough for all students to encode. In contrast, the embedded learning resource in the study described by Schwartz and Martin (2004) was sufficiently far from the classroom instruction. Therefore, only IPL students, who had acquired more flexible knowledge, could learn from it and apply the acquired knowledge successfully. This explanation further suggests that in the absence of additional instruction, only Full IPL students in our study could make the leap and answer the target new-strategy items.
While this argument explains performance on new-strategy items (with or without instruction) in terms of distance from classroom instruction, it does not explain what factors determine this distance. What makes some items ‘closer’ than others? What prepared Full IPL students for improved performance on some items but not on others?
Students may grapple with many challenges during the invention phase, many of which do not receive attention during classroom instruction. Students who invent are exposed to various challenges by virtue of attempting to invent general valid methods. We hypothesize that students use knowledge acquired during these experiences when later integrating new-strategy tasks into their existing body of knowledge. For example, the post-tests in this study included three new-strategy items, requiring the following new strategies: (1) comparing multiple datasets in a single representation; (2) representing data in unconventional intervals; and (3) finding the ratio between variability and average in order to account for differences in magnitude. These topics were not covered during classroom instruction. However, when we analyzed students’ inventions, we noticed that many inventions included features that could prepare students to expand the instructed knowledge and invent the first two strategies (see Figure 3). Subsequently, Full IPL students demonstrated better performance on the relevant new-strategy items. At the same time, no student attempted during invention to compare datasets with different magnitudes. Correspondingly, Full IPL students did not exhibit better performance on this new-strategy item.

=== Connections===

* Tim Nokes's study

=== Further Information ===

=== Annotated bibliography ===

[[Category:Empirical Study]]
[[Category:Protected]]

Direct instruction

2008-04-09T18:41:06Z

Idoroll:

[[category:glossary]]
[[category:IPL]]
[[category:Coordinative Learning]]

- this is only a stub -

Relevant knowledge explicitly delivered to students. Can include both procedural (cookbook) and conceptual components.
* Classroom instruction: mediated by the teacher
* Written instruction: Embedded within test or HW. Often includes a [[Worked examples]] component.
[[Category:Protected]]

IPL Instructional Principles

2008-04-09T18:40:53Z

Idoroll:

==Brief statement of principle==

Asking student to invent solutions to carefully designed challenges prior to receiving instruction can promote learning from subsequent instruction.

==Description of principle==
===Operational definition===
Students should attempt to rank alternatives in an [[invention task]] by [[comparing sets]] of contrasting cases, before receiving direct instruction and practice.

===Examples===
The following example is an [[invention task]] using [[comparing sets|set comparison]] as a preparation for learning about variance

NASA is about to launch its latest weather satellite to space, with the goal of monitoring global warming.
In order to put a satellite in orbit, it is being sent to space on a rocket. The rocket releases the satellite when it reaches it highest peak.
NASA considers three rockets for this task: Fly-I, Orbitter, and Icarus. Each rocket was tested 4-5 times. At this point NASA wants to choose one rocket for further development.

While the amount of fuel may not be the right one (that is, perhaps more or less fuel was needed), it was identical in all trials. This means that at this point NASA does not care about the absolute height, since the amount of fuel will need to be adjusted. But NASA does care about the ability to predict what height the rocket will reach – that it, how consistent the rocket is. A consistent rocket arrives at the same height every time.

Which rocket would you recommend?

The following graphs show the height the rockets reached during testing, relative to the desired height:

[[Image:NASA task.jpg]]

==Experimental support==

===Laboratory experiment support===
===In vivo experiment support===
Schwartz & Martin (2004) found that IPL activities help students learn from subsequent instruction.
[[Roll IPL|Roll, Aleven, Schwartz and Koedinger]] are currently carrying out another classroom study on the topic.

==Theoretical rationale==
(These entries should link to one or more [[:Category:Learning Processes|learning processes]].)

==Conditions of application==
Several conditions are being investigated in the current [[Roll IPL|IPL]] study, namely:
- The need for design in the invention process
- The need for debugging in the invention process

It is being hypothesizes, though has not been tested empirically yet, that the [[comparing sets|set comparison]] tasks should use contrasting cases and not isomorphic cases.

==Caveats, limitations, open issues, or dissenting views==
Several researchers object any form of discovery activity, and argue that [[direct instruction]] is always the superior alternative (Kirschner, P.A., Sweller, J., & Clark, J.E., 2004)

==Variations (descendants)==
==Generalizations (ascendants)==
==References==
* Kirschner, P.A., Sweller, J., & Clark, J.E. (2004). Why minimal guidance during instruction does not
work: An analysis of the failure of constructivist, Discovery, Problem-Based, Experiential, and
Inquiry-Based Teaching. Educational Psychologist, 41(2), 75–86.
* Schwartz, D. L., & Martin, T. Inventing to Prepare for Future Learning:
The hidden efficiency of encouraging original student production in statistics
instruction. Cognition and Instruction, 22(2), 2004, pp. 129-184
[[Category:Glossary]]
[[Category:Instructional Principle]]
[[Category:Protected]]

Roll IPL

2008-04-09T18:40:14Z

Idoroll:

Composition Effect Kao Roll - old, please keep

2008-04-09T18:40:01Z

Idoroll:

== The Composition Effect - What is the Source of Difficulty in Problems which Require Application of Several Skills? ==
Ido Roll, Yvonne Kao, Kenneth E. Koedinger

=== Abstract ===

Composite problems, i.e., problems that require the application of more than one skill, are shown to be harder than a collection of single-step problems requiring the application of the same set of skills.
A common explanation is that the composition itself imposes another difficulty level. However, an alternative explanation suggests that the composition makes the application of the individual skills harder. According to that explanation, poor feature validity and shallow domain rules make it harder on students to apply the individual skills correctly in the cluttered environment of composite problems, regardless the need to apply additional skills.
Our study investigates these issues in two ways: (1) Having a DFA which evaluates performance on composite problems and single-step problems using the same data, and (2) By evaluating the effect of instruction targeting a common misconception in single-step problems on composite problems.

=== Glossary ===

- Composite problems: Problems which require the application of several skills, such as solving 3x+6=0 for x.

- Single-step problems: Problems which require the application of a single skill, such as y+6=0 or 3x=-6

- DFA (Difficulty Factor Analysis): A test that includes pairs of items varying along one dimension only. It allows to evaluate the difficulty level of the single dimensions along which the problems differ.

- The Composition Effect: The effect according to which composite problems are harder than a set of single-step problems using the same skills.

=== Research question ===

What is the main source of difficulty in composite problems?

=== Background and Significance ===

Significance: This study can shed some light on the source of difficulty on composite problems, and thus can inform the design of relevant instruction and remediation.

=== Independent Variables ===

An instruction in the form of solved-example, targeting a common misconception - identifying base and hight in a cluttered environment.

=== Dependent variables ===

Three tests are used in the study:
- Pre-test: given before all instruciton
- Mid-test: given after students learned about single-step problems and before composite problems
- Post-test: after students have learned and practice all material.

The tests include the following items. Some of which are [[transfer]] items, evaluating robust learning, since they require and adaptive application of the knowledge learned and practiced in class.

* Simple diagram:
*# no distractors, canonical orientation
*# distractors, canonical orientation
*# no distractors, tilted orientation
*# distractors, tilted orientation
* Complex diagram:
*# Given complex diagram, ask for skill A
*# Given complex diagram, ask for skill B
*# Given steps A and B, ask for skills C (which requires A and B)
*# Given complex diagram, ask for C (which requires A and B)

=== Hypothesis ===

# The difficulty level in composite problems originates in poor feature validity of the single skills.
#* An operationalized version of this hypothesis is that performance on items of type "Find measure C based on diagram" will be equivalent to the multiplication of success rate on items "Find measure C based on items A and B", "Find measure A based on diagram", and "Find measure B based on diagram".
# Tilted orientation and distractors still imposes difficulty even once students mastered the skills
# Direct instruction during the test which targets these misconceptions in the form of a solved example can improve performance

=== Findings ===

None yet.

=== Explanation ===

=== Descendents ===

=== Annotated bibliography ===

Bransford (2000). How people learn: brain, mind, experience, and school National Academy Press.
Heffernan, N.T., & Koedinger, K.R. (1997) The composition effect in symbolizing: The role of symbol production vs. text comprehension. in proceedings of Nineteenth Annual Conference of the Cognitive Science Society, 307-12. Hillsdale, NJ: Erlbaum.
Koedinger, K.R., & Anderson, J.R. (1997). Intelligent Tutoring Goes to School in the Big City. International Journal of Artificial Intelligence in Education 8, 30-43
Koedinger, K. R. & Cross, K. (2000). Making informed decisions in educational technology design: Toward meta-cognitive support in a cognitive tutor for geometry. Presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
Owen, E., & Sweller, J. (1985). What do students learn while solving mathematics problems? Journal of Educational Psychology, 77, 272-284.
Simon, H. A., & Lea, G. (1974). Problem solving and rule induction: A unified view. In L. W. Gregg (Ed.), Knowledge and cognition. Hillsdale, NJ: Erlbaum.

[[Category:Empirical Study]]
[[Category:Protected]]

Help Lite (Aleven, Roll)

2008-04-09T18:39:47Z

Idoroll:

== Hints during tutored problem solving – the effect of fewer hint levels with greater conceptual content ==
Vincent Aleven, Ido Roll, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll

Other Contributers: Ron Salden (post-doc)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 5/2006 || 5/2006 || Wilkinsburg (Geometry) || 40 || 90 || Yes
|}

=== Abstract ===

This in vivo experiment compared the effectiveness of two styles of hint sequences during tutored problem solving. The study was carried out in the Geometry LearnLab.
Two conditions were compared, each working with its own tutor version. The tutor versions differed only with respect to the content of the hint sequences. A key difference between the hint sequences was that the number of hint levels was reduced from about 7 in a typical hit sequence to 2 or 3. This was achieved by removing hints that merely reminded students of their current goal within the problem, by removing hints that encouraged students to try to address their question by using the Glossary, and by being more concise in explaining how a theorem or definition could be explained. At the same time, conceptual content was added, in the form of explanations of geometry terms.

=== Glossary ===

* [[help-seeking behavior]]

* [[Clicking through hints]]

* [[Hint sequence]]

* [[Help avoidance]]

* [[Help abuse]]

* [[Bottom out hint]]

* [[Metacognition]]

* [[Game the system]]

* [[Cognitive tutor]]

=== Research question ===

How is robust learning affected by shorter hint sequences with richer conceptual content?

This was not an independent study, but part of the main [[The Help Tutor Roll Aleven McLaren | Help Seeking]] study.

=== Background and Significance ===

=== Independent Variables ===

Number and type of hint levels within the [[hint sequence]]:
* Control: Standard Cognitive Tutor hints; Including 7 levels of hints of different types: containing either domain knowledge or metacognitive hints (such as 'search the glossary for ...')
* Experimental condition: Included only 2-3 levels of hints, each of which includes only domain knowledge.

=== Dependent variables ===

The study uses two levels of dependent measures:

Assessing Help Seeking behavior:
* Analyzing log-files against a model of ideal help-seeking behavior

Assessing domain learning
* Learning curves while using the tutor
* % correct on attempts following hint requests

Due to technical and administrative errors, some of the tests are lost, and others cannot be attributed to conditions. As a result, no pre- and post-test measures can be used.

=== Hypothesis ===

Students pay more attention to short hint sequences as they feel they are more helpful and easier to understand. Thus, the shorter hint sequences reduce hint abuse, such as students’ clicking through hints until they get the answer, without paying attention to why the answer is what it is. The richer conceptual content helps them to make sense out of the tutor’s hints, reducing implicit learning and also making students more likely to attend to the hints. Thus, there are two reasons why the new hints result in better sense making and less implicit learning.

=== Findings ===

None. Errors in data logging and data collection do not allow for an extensive analysis.

=== Explanation ===

Having informative, relevant and on-time hints provides the student an effective learning trajectory when learning-by-doing becomes to difficult.
The original help sequence require the learner for more responsibility - identify relevant hints, search the glossary, etc. These activities require [[cognitive load]]. However, the updated [[hint sequence]] offer relevant instruction when required, and low on extraneous [[cognitive load]].

=== Descendents ===

=== Annotated bibliography ===

Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag.

Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th Int C on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag.

Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th Int C on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press.

Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int J of Artificial Intelligence in Education(16), 101-30

Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th Int C on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag.

Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag.

Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th Int C on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag.

Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag.

[[Category:Empirical Study]]
[[Category:Protected]]

IPL Instructional Principles

2008-03-25T16:57:56Z

Idoroll: /* Examples */

IPL Instructional Principles

2008-03-25T16:57:31Z

Idoroll: /* Examples */

File:NASA task.jpg

2008-03-25T16:56:39Z

Idoroll: