Theory Wiki - User contributions [en]

File:Tutor w hint.jpg

2008-02-13T19:50:05Z

Gurpreet:

Help facilities

2008-02-13T19:49:35Z

Gurpreet:

Help Facilities: The means available to the student, which offer the student support in the learning process. These include:
* Human help resources: Usually, students can consult with their teacher and peers.
* Online resources: Online help resources are given either on-demand or proactively, and are usually contextualized or non-contextualized. The table below demonstrates these categories using the Help Seeking facilities of the Geometry Cognitive Tutor, as used in the Help Tutor project:
{| cellpadding="20" cellspacing="0" border="1"
|
! Contextual
! Non contextual
|-
! On demand
| Student can ask for contextual hints, relevant to the specific step she attempts to solve. Evey hint has several levels the students can browse, with the most elaborated one conveying the answer.
| The Glossary offers students a de-contextualized help resource, similar to an online dictionary. Students can find relevant information by searching it.
|-
! Proactive
| When students repeat an error more than a predefined number of times, the system presents them automatically with the hint they would have gotten were they to ask for one.
| The Cognitive Tutor does not have a proactive non-contextual hint. It is unclear how such hint will look like.
|}

[[Image:tutor_w_hint.jpg]]

For a comprehensive review of help in online system, see
Aleven, V., Stahl, E., Schworm, S., Fischer, F., & Wallace, R.M. (2003). Help Seeking and Help Design in Interactive Learning Environments. Review of Educational Research, 73(2), 277-320.

[[Category:Glossary]]
[[Category:Interactive Communication]]
[[Category:Help Tutor]]

File:Roll Pyramid.jpg

2008-02-13T19:44:27Z

Gurpreet:

File:Help-seeking and learning.jpg

2008-02-13T19:40:26Z

Gurpreet:

File:Help-seeking behavior.jpg

2008-02-13T19:38:56Z

Gurpreet:

The Help Tutor Roll Aleven McLaren

2008-02-13T19:38:04Z

Gurpreet: /* Findings */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll_Pyramid.jpg]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking_and_learning.jpg]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Log files from study 1 were analyzed to evaluate whether students improved their help-seeking behavior while working with the Help Tutor. While overall there was only a minor decrease in the error rate (from 19% in the control condition to 16% in the Help Tutor condition), there was a significant decrease in the error rate on first hints, following hints, and the portion of bottom-out hints out of all hint-sequences.

[[Image:help-seeking_behavior.jpg]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

To evaluate whether the effect of the help-seeking curriculum persists beyond the tutored environment, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., & Koedinger, K.R. (To appear) Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring. To appear in the proceedings of the International Conference on Artificial Intelligence in Education 2007.
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

File:Study 1 results.jpg

2008-02-13T19:36:59Z

Gurpreet:

File:Study 2 results.jpg

2008-02-13T19:35:56Z

Gurpreet:

The Help Tutor Roll Aleven McLaren

2008-02-13T19:33:07Z

Gurpreet: /* Evaluation of goal 3: Improve domain learning */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll Pyramid.png]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking and learning.png]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Log files from study 1 were analyzed to evaluate whether students improved their help-seeking behavior while working with the Help Tutor. While overall there was only a minor decrease in the error rate (from 19% in the control condition to 16% in the Help Tutor condition), there was a significant decrease in the error rate on first hints, following hints, and the portion of bottom-out hints out of all hint-sequences.

[[Image:help-seeking behavior.png]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study_1_results.jpg]]

Study 2 results:

[[Image:study_2_results.jpg]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

To evaluate whether the effect of the help-seeking curriculum persists beyond the tutored environment, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., & Koedinger, K.R. (To appear) Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring. To appear in the proceedings of the International Conference on Artificial Intelligence in Education 2007.
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

File:Declarative knowledge.jpg

2008-02-13T19:32:08Z

Gurpreet:

The Help Tutor Roll Aleven McLaren

2008-02-13T19:31:56Z

Gurpreet: /* Evaluation of goal 4: Improve future metacognitive behavior */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll Pyramid.png]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking and learning.png]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Log files from study 1 were analyzed to evaluate whether students improved their help-seeking behavior while working with the Help Tutor. While overall there was only a minor decrease in the error rate (from 19% in the control condition to 16% in the Help Tutor condition), there was a significant decrease in the error rate on first hints, following hints, and the portion of bottom-out hints out of all hint-sequences.

[[Image:help-seeking behavior.png]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study 1 results.png]]

Study 2 results:

[[Image:study 2 results.png]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

To evaluate whether the effect of the help-seeking curriculum persists beyond the tutored environment, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative_knowledge.jpg]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., & Koedinger, K.R. (To appear) Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring. To appear in the proceedings of the International Conference on Artificial Intelligence in Education 2007.
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

The Help Tutor Roll Aleven McLaren

2008-02-13T19:30:54Z

Gurpreet: /* Evaluation of goal 4: Improve future metacognitive behavior */

== Towards Tutoring [[Metacognition]] - The Case of Help Seeking ==
Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

=== Meta-data ===

PI's: Vincent Aleven, Ido Roll, Bruce M. McLaren

Other Contributers: EJ Ryu (programmer)
{| border="1"
! Study # !! Start Date !! End Date !! LearnLab Site !! # of Students !! Total Participant Hours !! DataShop?
|-
| '''1''' || 2004 || 2004 || Analysis of existing data || 40 || 280 || No, old data
|-
| '''2''' || 2005 || 2005 || Analysis of existing data || 70 || 105 || No, old data
|-
| '''3''' || 5/2005 || 5/2005 || Hampton & Wilkinsburg (Geometry) || 60 || 270 || No, incompatible format
|-
| '''4''' || 2/2006 || 4/2006 || CWCTC (Geometry) || 84 || 1,008 || Yes
|}

=== Abstract ===

While working with a tutoring system, students are expected to regulate their own learning process. However, often, they demonstrate inadequate [[metacognition|metacognitive process]] in doing so. For example, students often ask for help too frequently or not frequently enough.
In this project we built an Intelligent Tutoring System to teach [[metacognition]], and in particular, to improve students' [[help-seeking behavior]]. Our Help Seeking Support Environment includes three components:
# Direct [[help-seeking behavior|help seeking]] [[explicit instruction|instruction]], given by the teacher
# A [[Self-Assessment]] Tutor, to help students evaluate their own need for help
# The Help Tutor - a domain-independent agent that can be added as an adjunct to a [[cognitive tutor]]. Rather than making help-seeking decisions for the students, the Help Tutor teaches better help-seeking skills by tracing students actions on a (meta)cognitive [[help-seeking model]] and giving students appropriate feedback.

In a series of [[in vivo experiment]]s, the Help Tutor accurately detected help-seeking errors that were associated with poorer learning and with poorer [[declarative]] and [[procedural]] [[knowledge component]]s of help seeking. The main findings were that students made fewer help-seeking errors while working with the Help Tutor and acquired better help seeking [[declarative]] [[knowledge component]]s.
However, we did not find evidence that this led to an improvement in learning at the domain level or to better [[help-seeking behavior]] in a paper-and-pencil environment.
We pose a number of hypotheses in an attempt to explain these results. We question the current focus of metacognitive tutoring, and suggest ways to reexamine the role of [[help facilities]] and of metacognitive tutoring within Intelligent Tutoring Systems.

=== Background and Significance ===

Not only that teaching [[metacognition]] holds the promise of improving current learning of the domain of interest, but also, or even mainly, it can accelerate future learning and successful regulation of independent learning. One example to metacognitive knowledge is help-seeking [[knowledge component]]s: The ability to identify the need for help, and to elicit appropriate assistance from the [[relevant resources|help facilities].
However, considerable evidence shows that metacognitive [[knowledge component]]s are in need of better support. For example, while working with Intelligent Tutoring Systems, students try to "[[game the system]]" or do not [[self-explanation|self-explain]] enough. Similarly, research shows that students' [[help-seeking behavior]] leaves much room for improvement.

==== Shallow help seeking [[knowledge component]]s ====
Research shows that students do not use their help-seeking konwledge components approrpiately. For example, Aleven et al. (2006) show that 30% of students' actions were consecutive fast help requests (a common form of [[help abuse]], termed '[[clicking through hints]]'), without taking enough time to read the requested hints.
Extensive log-file analysis suggests that students apply faulty [[knowledge component]]s such as the following:

'''Faulty [[procedural]] [[knowledge component]]s:'''
Cognitive aspects:
If I don’t know the answer =>
I should guess

Motivational aspects:
If I get the answer correct =>
I achieved the goal

Social aspects:
If I ask for help =>
I am weak

'''Faulty [[declarative]] [[knowledge component]]s:'''

Asking for hints will always reduce my skill level

Making an error is better than asking for a hint

Only weak people ask for help

==== Teaching vs. supporting [[metacognition]] ====

Several systems support students' metacognitive actions in a way that encourages, or even forces, students to learn productively and efficiently. For example, a tutoring system can require the student to self-explain. While this approach is likely to improve domain learning in the supported environment, the effect is not likely to persist beyond the scope of the tutoring system, and therefore is not likely to help students become better future learners.

Towards that end, we chose not to '''support''' students' help seeking actions, but to '''teach''' them better help-seeking skills. Rather than making the metacognitive decisions for the students (for example, by preventing help-seeking errors or gaming opportunities), this study focuses on helping students refine their Help Seeking [[knowledge component]]s and acquire better [[feature validity]] of their [[help-seeking behavior|help-seeking]] [[metacognition|metacognitive skills]].

By doing so, we examine whether metacognitive knowledge can be taught using familiar conventional domain-level pedagogies.

=== Glossary ===
See [[:Category:Help Tutor|Help Tutor Glossary]]

=== Research questions ===

# Can conventional and well-established instructional principles in the domain level be used to tutor [[metacognition|metacognitive]] [[knowledge component]]s such as [[help-seeking behavior|Help Seeking]] [[knowledge component]]s?
# Does the practice of better metacognitive behavior translates, in turn, to better domain learning?

In addition, the project makes the following contributions:
# An improved understanding of the nature of help-seeking knowledge and its acquisition.
# A novel framework for the design of goals, interaction and assessment for metacognitive tutoring.

=== Independent Variables ===

Two studies were performed with the Help Tutor. In both studies the independent variable was the existence of help seeking support.
Control condition used the conventional Geometry Cognitive Tutor

[[Image:Geometry Cognitive Tutor.jpg]]

The treatment condition varied between studies:
* Study one: The Geometry Cognitive Tutor + the Help Tutor
* Study two: The Geometry Cognitive Tutor + the Help Seeking Support Environment (help seeking explicit instruction, self-assessment tutor, and Help Tutor)

'''The Help Tutor:'''
The Help Tutor is an a Cognitive Tutor in its own right that identifies recommended types of actions by tracing students’ interaction with the Geometry Cognitive Tutor relative to a metacognitive help-seeking model. When students perform actions that deviate from the recommended ones, the Help Tutor presents a message that stresses the recommended action to be taken. Messages from the metacognitive Help Tutor and the domain-level Cognitive Tutor are coordinated, so that the student receives only the most helpful message at each point [2].

[[Image:The Help-tutor.jpg]]
[[image:The Help Seeking Model.jpg]]

'''The Self-Assessment Tutor:'''
The ability to correctly self-assess one’s own knowledge level is correlated with strategic use of help (Tobias and Everson, 2002). The Self-Assessment Tutor is designed to tutor students on
their self-assessment skills; to help students make appropriative learning decisions based on their self assessment; and mainly, to give students a tutoring environment, low on cognitive load, in which they can practice using their help-seeking skills.
The curriculum used by the Treatment group in study two consists of interleaving Self Assessment and Cognitive Tutor + Help Tutor sessions, with the Self Assessment sessions taking about 10% of the students’ time. During each self-assessment session the student assesses the skills to be practiced in the subsequent Cognitive Tutor section.

[[Image:The Self-Assessment Tutor.jpg]]

'''Explicit help-seeking instruction:'''
As White and Frederiksen demonstrated (1998), reflecting in the classroom environment on the desired metacognitive process helps students internalize it. With that goal in mind, we created a short classroom lesson about help seeking with the following objectives: to give students a better declarative understanding of desired and effective help-seeking behavior; to improve their dispositions and attitudes towards seeking help; and to frame the help-seeking knowledge as an important learning goal, alongside Geometry knowledge, for the coming few weeks. The instruction includes a video presentation with examples of productive and faulty help-seeking behavior and the appropriate help-seeking principles.

[[Image:Explicit help-seeking instruction.jpg]]

=== Dependent variables ===

The study uses two levels of dependent measures:
# Directly assessing Help Seeking skills
# Assessing domain-level learning, and by that evaluating the contribution of the help-seeking skills.

1. Assessments of help-seeking konwledge:
* [[Normal post-test]]:
** Declarative: hypothetical help-seeking dilemmas
** Procedural: Help seeking error rate while working with the tutor
* [[Transfer]]: Ability to use optional hints embedded within certain test items in the paper test.

[[Image:embedded hints.jpg]]

2. Assessments of domain konwledge:
* [[Normal post-test]]: Problem solving and explanation items like those in the tutor's instruction.
* [[Transfer]]:
** Data insufficiency (or "not enough information") items.
** Conceptual understanding items (study two only)

=== Hypothesis ===

The combination of explicit help-seeking instruction, on-time feedback on help seeking errors, and raising awareness to knowledge deficits will
* Improve [[feature validity]] of students' help seeking skills
and thus, in turn, will
* Improve learning of domain knowledge by using those skills effectively.

==== Instructional Principles ====

The main principles being evaluated here is whether [[Roll_help seeking principle | instruction should support meta-cognition in the context of problem solving]] by using principles of cognitive tutoring such as:
* Giving direct instruction
* Giving immediate feedback on errors
* Prompting for self-assessment

This utilizes the following instructional principles:

* The Self-Assessment Tutor utilizes the [[Reflection questions]] principle
* The Help Tutor itself utilizes the [[Tutoring feedback]] principle
* The Help Seeking Instruction utilizes the [[Explicit instruction]] principle.

=== Findings ===

As seen below (adapted from Roll et al. 2006), metacognitive tutoring has the following goals:
# First, the tutoring system should capture metacognitive errors (in our case, help-seeking errors).
# Then, it should lead to an improved metacognitive behavior within the tutoring system.
# This, in turn, should lead to an improvement in the domain learning.
# The effect should persist beyond the scope of the tutoring system.
# As a result, students are expected to become better future learners.

[[Image:Roll Pyramid.png]]

==== Evaluation of goal 1: Capture metacognitive errors ====

In study 1, 17% of students actions were classified as errors. These errors were singificantly negatively correlated with learning (r=-0.42) - the more help-seeking errors captured by the system, the smaller the improvement from pre- to post-test.
This data suggests that the help-seeking model captures appropriate actions, and that the goal was achieved - the Help Tutor captures help-seeking errors.

[[Image:help-seeking and learning.png]]

==== Evaluation of goal 2: Improve metacognitive behavior ====

Log files from study 1 were analyzed to evaluate whether students improved their help-seeking behavior while working with the Help Tutor. While overall there was only a minor decrease in the error rate (from 19% in the control condition to 16% in the Help Tutor condition), there was a significant decrease in the error rate on first hints, following hints, and the portion of bottom-out hints out of all hint-sequences.

[[Image:help-seeking behavior.png]]

This data suggests that while the system was not able to improve all aspects of the desired metacognitive behavior, it did improve students' behavior on the common types of errors.

==== Evaluation of goal 3: Improve domain learning ====

While students' help seeking behavior improved while working with the Help Tutor (in study 1) or the full Help Seeking Support Environment (in study 2), we did not observe differences in learning between the two conditions on neither study 1 nor study 2.

Study 1 results:

[[Image:study 1 results.png]]

Study 2 results:

[[Image:study 2 results.png]]

* Notice, since the tests in times 1 and 2 evaluated different instructional units (Angles vs. Quadrilaterals), lower grades at time 2 do not suggest decrease in knowledge.

==== Evaluation of goal 4: Improve future metacognitive behavior ====

To evaluate whether the effect of the help-seeking curriculum persists beyond the tutored environment, students' help seeking behavior was evaluated in a transfer environment - the paper and pencil tests.

Hypothetical help seeking dilemmas, such as the one described below, were used to evaluate declarative help-seeking knowledge.

1. You tried to answer a question that you know, but for some reason the tutor says that your answer is wrong. What should you do?
[ ] First I would review my calculations. Perhaps I can find the mistake myself?
[ ] The Tutor must have made a mistake. I will retype the same answer again.
[ ] I would ask for a hint, to understand my mistake.

Procedural help-seeking skills were evaluated using embedded hints in the tests (see figure in the Dependant Measures section above).

In study 1 (which included only the Help Tutor component,) students in the Treatment condition demonstrated neither better declarative nor procedural help-seeking knowledge, compared with the Control condition.

In study 2 (which included the explicit help-seeking instruction and the Self-Assessment tutor in addition to the Help Tutor) students in in the Treatment condition demonstrated better declarative help-seeking knowledge (compared with Control group students) but no better procedural knowledge

[[Image:Declarative knowledge.png]]
[[Image:Procedural_knowledge.jpg]]

==== Evaluation of goal 5: Improve future domain learning ====

Due to technical difficulties, this goal was not evaluated in both studies.

==== Summary of results ====

Overall, the following pattern of results emerges from the studies:
- The Help Seeking Support Environment intervened appropriate actions during the learning process
- Students improved their help-seeking behavior while working with the system
- Students acquired better help-seeking declarative knowledge following the system
- However, students' domain learning did not improve
- Also, the improvement in students' help seeking behavior did not persist beyond the tutoring system.

=== Explanation ===

These somewhat disappointing results raise an important question: Why did the environment not lead to an improvement in learning and in help-seeking behavior in the paper-test measures? Why did the improved online help-seeking behavior not lead to improved learning gains?

==== Hypothesis 1: Students do not have the skills, but we didn't teach them right. ====
One possible explanation may be that the Help Seeking Support Environment requires excessive cognitive load during problem solving. Clearly, the learning process with the Help Seeking Support Environment is more demanding compared with the conventional Cognitive Tutor alone, since more needs to be learned. However, much of the extra content is introduced during the classroom discussion and self-assessment sessions. The only extra content presented during the problem-solving sessions are the Help Tutor’s error messages, but they are not expected to increase the load much, especially given that a prioritization algorithm makes sure students receive only one message at a time (either from the Help Tutor or the [[cognitive tutor]]). Also, if the Help Seeking Support Environment indeed creates requires too much cognitive load, it should be expected to hinder learning, which we did not observe.

==== Hypothesis 2: The role of help seeking in ITS ====
Hints in tutoring systems have two objectives: to promote learning of challenging skills, and to help students move forward within the curriculum (i.e., to prevent them from getting stuck). While the latter is achieved easily with both the Cognitive Tutor and the Help Seeking Support Environment, achieving the first is much harder. It is not yet clear what makes a hint into a good hint, and how to create an effective [[hint sequence]]. It is possible that the hints, as implemented in the units of the Cognitive Tutor we used, are not optimal. For example, there may be too many levels of hints, with each level adding too little information to the previous one. Also, perhaps the detailed explanations are too demanding with regard to students’ reading comprehension ability. It is quite possible that these hints, regardless of how they are being used, are above students' [[zone of proximal development]], and thus do not contribute much to learning. Support for that idea comes from Schworm and Renkl (2002). They found that offering explanations by the system impaired learning when [[self-explanation]] was required. The Geometry Cognitive Tutor prompts for [[self-explanation]] in certain units. Perhaps elaborated hints are redundant, or even damaging, when [[self-explanation]] is required.
It is possible also that [[help-seeking behavior]] that we currently view as faulty may actually be useful and desirable, in specific contexts for specific students. For example, perhaps a student who does not know the material should be allowed to view the bottom-out hint immediately, in order to turn the problem into a solved example. Support for that idea can be found in a work by Yudelson et al. (2006), in which medical students in a leading med school successfully learned by repeatedly asking for more elaborated hints. Such “clicking-though hints” behavior would be considered faulty by Help Seeking Support Environment. However, this population of students is known to have good metacognitive skills (without them it is unlikely they would have reached their current position). Thus, it seems that sometimes “misusing” help (e.g., [[help abuse]], according to the Help Seeking Support Environment) can be beneficial to some students. Further evidence can be found in Baker et al. (2004), who showed that some (but not all) students who “[[game the system]]” (i.e., click through hints or guess repeatedly) learn just as much as students who do not game. It may be the case that certain gaming behaviors are adaptive, and not irrational. Students who use these strategies will insist on viewing the [[bottom out hint]] and will ignore all intermediate hints, whether domain-level or metacognitive. Once intermediate hints are ignored, better [[help-seeking behavior]] according to the Help Seeking Support Environment should have no effect whatsoever on domain [[knowledge component]]s, as indeed was seen.
It is possible that we are overestimating students’ ability to learn from hints. Our first recommendation is to re-evaluate the role of hints in [[cognitive tutor]]s using complementary methodologies such as log-file analysis (e.g., Chang (2006) uses dynamic Bayes nets to evaluate the contribution of hints in a reading tutor); tracing individual students (to evaluate the different uses students make of hints), experimentation of different types of hints (for example, proactive vs. on demand), and analysis of human tutors who aid students while working with [[cognitive tutor]]s.

==== Hypothesis 3: The focus of metacognitive tutoring in ITS. ====
The previous hypothesis, focused on students’ tendency to skip hints, suggests that perhaps the main issue is not lack of knowledge, but lack of motivation. In other words, '''perhaps students already have these skills in place, but choose not to use them.''' For students who ignore intermediate hints, metacognitive messages offer little incentive. While the Help Seeking Support Environment can increase the probability that a proper hint level appears on the screen, it has no influence on whether it is being read, or whether the student attempts to understand it. Students may ignore the messages for several reasons. For example, they may habitually click through hints, and may resent the changes that the Help Seeking Support Environment imposes. This idea is consistent with the teachers’ observation that the students were not fond of the Help Seeking Support Environment error messages. They may comply with them, in order to make progress, but beyond that will ignore their content. The test data discussed above provides support for this idea. On 7 out of the 12 hint evaluations (seen in the findings section of goal 4) students scored lower on items with hints than on items with no hints. A [[cognitive headroom]] explanation does not account for this difference, since the Request hints did not add much load. A more likely explanation is that students chose to skip the hints since they were new to them in the given context. Baker (2005) reviewed several reasons for why students [[game the system]]. While no clear answer was given, the question is applicable here as well.
Motivational issues bring us to our final hypothesis. Time preference discount (Feldstein, 1964) is a term coined in economics, that describes behavior in which people would rather have a smaller immediate reward over a distant greater reward. In the tutoring environment, comparing the benefit of immediate correct answer with the delayed benefit (if any) of acting metacognitively correct may often lead the student to choose the first. If that is indeed the case, then students may already have the right metacognitive skills in place. The question we should be asking ourselves is not only how to get students to learn the desired metacognitive skills – but mainly, how to get students to use them.

=== Connections===

* The Help Tutor attempts to extend traditional tutoring beyond the common domains. In that, it is similar to the work of Amy Ogan on tutoring [[FrenchCulture | French Culture]]

* The manipulation of interaction between the student and the tutor, which is "natural" in the control condition, is guided by the help tutor. This is similar to the scripting manipulation of the [[Rummel Scripted Collaborative Problem Solving]] and the [[Walker A Peer Tutoring Addition]] projects.

* Another example for studying the effects of hints is [[Ringenberg Examples-as-Help|Ringenberg's study]], in which hints are compared to examples.

* Going to do an in-vivo study at a LearnLab site? Check out how to answer [[FAQ for teachers|teacher's FAQ]]

=== Further Information ===
Plans for June 2007 - Dec. 2007:
* Present the study in the International Conference on Artificial Intelligence on Education
* Submit camera ready copy of the paper to the Journal on Metacognition and Instruction
* Analyze the logfiles for Metacognitive learning

=== Annotated bibliography ===

# Aleven, V., & Koedinger, K.R. (2000) Limitations of student control: Do students know when they need help? in proceedings of 5th International Conference on Intelligent Tutoring Systems, 292-303. Berlin: Springer Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2004) Toward tutoring help seeking - Applying cognitive modeling to meta-cognitive skills . in proceedings of 7th International Conference on Intelligent Tutoring Systems, 227-39. Berlin: Springer-Verlag. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., & Koedinger, K.R. (2005) An architecture to combine meta-cognitive and cognitive tutoring: Pilot testing the Help Tutor. in proceedings of 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands: IOS press. [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Aleven, V., McLaren, B.M., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. Int Journal of Artificial Intelligence in Education(16), 101-30 [[http://www.cs.cmu.edu/~aleven/publications.html#help-seeking pdf]]
# Baker, R.S., Corbett, A.T., & Koedinger, K.R. (2004) Detecting Student Misuse of Intelligent Tutoring Systems. in proceedings of 7Th International Conference on Intelligent Tutoring Systems, 531-40.
# Baker, R.S., Roll, I., Corbett, A.T., & Koedinger, K.R. (2005) Do Performance Goals Lead Students to Game the System? in proceedings of 12Th International Conference on Artificial Intelligence in Education, 57-64. Amsterdam, The Netherlands: IOS Press.
# Chang, K.K., Beck, J.E., Mostow, J., & Corbett, A. (2006) Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. in proceedings of Workshop on Educational Data Mining at AAAI 2006, 41-6. Menlo Park, California: AAAI.
# Feldstein, M.S. (1964). The Social Time Presence Discount Rate in Cost-Benefit Analysis. The Economic Journal 74(294), 360-79
# Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., & Koedinger, K.R. (2005) Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. in L. Ardissono, (Eds.), in proceedings of User Modeling 2005, 379-88. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Ryu, E., Sewall, J., Leber, B., McLaren, B.M., Aleven, V., & Koedinger, K.R. (2006) Towards Teaching Metacognition: Supporting Spontaneous Self-Assessment. in proceedings of 8th International Conference on Intelligent Tutoring Systems, 738-40. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S., & Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improves Students' Help-Seeking Actions, Skills and Learning? in proceedings of 8th Int C on Intelligent Tutoring Systems, 360-9. Berlin: Springer Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]
# Roll, I., Aleven, V., McLaren, B.M., & Koedinger, K.R. (To appear) Can Help Seeking Be Tutored? Searching for the Secret Sauce of Metacognitive Tutoring. To appear in the proceedings of the International Conference on Artificial Intelligence in Education 2007.
# Schworm, S., & Renkl, A. (2002) Learning by solved example problems: Instructional explanations reduce self-explanation activity. in proceedings of The 24Th Annual Conference of the Cognitive Science Society, 816-21. Mahwah, NJ: Erlbaum.
# Yudelson, M.V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R.S. (2006) Mining Student Learning Data to Develop High Level Pedagogic Strategy in a Medical ITS. in proceedings of Workshop on Educational Data Mining at AAAI 2006, Menlo Park, CA: AAAI.# Roll, I., Aleven, V., & Koedinger, K.R. (2004) Promoting Effective Help-Seeking Behavior through Declarative Instruction. in proceedings of 7th International Conference on Intelligent Tutoring Systems, 857-9. Berlin: Springer-Verlag. [[http://www.andrew.cmu.edu/user/iroll/Publications.html pdf]]

[[Category:Empirical Study]]

File:Procedural knowledge.jpg

2008-02-13T19:30:41Z

Gurpreet:

File:Interest combined.jpg

2008-02-13T19:27:27Z

Gurpreet:

File:Graph40.jpg

2008-02-13T19:24:27Z

Gurpreet:

REAP Study on Personalization of Readings by Topic (Fall 2006)

2008-02-13T19:21:50Z

Gurpreet: /* Findings */

== REAP Study on Personalization of Readings for Increased Interest ==

=== Logistical Information ===

{| border="1"
|+
|-
| '''Contributors''' || Maxine Eskenazi, Alan Juffs, Michael Heilman, Kevyn Collins-Thompson, Lois Wilson, Jamie Callan
|-
| '''Study Start Date''' || September 11, 2006
|-
| '''Study End Date''' || November 21, 2006
|-
| '''Learnlab Courses''' || English Language Institute Reading 4 (ESL LearnLab)
|-
| '''Number of Students''' || 35
|-
| '''Total Participant Hours (est.)''' || 270
|-
| '''Data in Datashop''' || no
|}

=== Abstract ===

This paper discusses the enhancement of the REAP tutor to allow for [[personalization]] of reading materials
by topic in order to increase interest and motivation. In this work, the term “[[personalization]]” refers to the
selection of practice readings in order to match a student’s interests.

During each training session with REAP, students work through a series of readings, each of which is followed by
practice exercises for the target words in the reading. While reading a passage, students are able to access
dictionary definitions for any word in a reading either by clicking on a highlighted target word or by typing a
word into a box in the lower-left corner of the screen. The target words in the readings are also highlighted
because highlighting may increase the use of dictionary definitions, thus encouraging students to
coordinate multiple sources of information about a word’s meaning—namely, the implicit context around
words and the explicit definitions of words.

A problem discovered in past studies with REAP is that many students spend only a brief amount of time
on a reading and do not deeply process the text. Students often only read the dictionary definition for target
words rather than attempting to process the entire context around the words. Inferring the meaning of
vocabulary from context is a seemingly important strategy that is not used by such students. This behavior is likely due to a desire to perform well on post-reading practice exercises and post-test, which can be viewed as forms of extrinsic motivation. Intrinsically
motivated students who are more interested in a reading are more likely to read the entire text and to use
context to learn the meaning of unknown vocabulary. Therefore, [[personalization]] that increases intrinsic
motivation could lead to deeper processing of context and better learning of vocabulary.

{| border="1"
|+
|-
| ||'''Passive''' || '''Active''' || '''Interactive'''
|-
| '''Explicit (general)''' || Dictionary Definitions || || Practice Exercises
|-
| '''Implicit (instance)''' || Interpreting meaning in context while reading || Sentence Production (assessment) || Practice Exercises
|}

=== Glossary ===

''Intrinsic Motivation:'' Motivation to learn for learning's own sake rather than some external goal.

''Extrinsic Motivation:'' Motivation for learn in order to satisfy an external goal, such as completing a task or passing an assessment.

=== Research question ===

Do the benefits of [[personalization]] of practice readings by topics of interest outweigh the costs in a tutoring system for ESL vocabulary practice?

=== Dependent variables ===
[[Normal post-test]] scores

[[Normal post-test]] scores for practiced words only

[[Long-term retention]] test scores, same post-test but administered months later.

Evidence of [[Transfer]]: sentence production tasks for target words, correct use of words in writing assignments for other courses.

=== Independent variables ===
[[Personalization]] of readings by topics of interest. In the control condition, the tutor did not use potential personal interest as a factor in its selection of reading materials. In the treatment condition, the tutor did use interest as a factor. All other selection criteria were the same in both conditions. Time on task was also the same.

=== Hypotheses ===

Since intrinsic motivation seems to be important in language learning, the benefits of [[personalization]] will outweigh the costs.

=== Findings ===

Students in the treatment condition with [[personalization]] performed better on average (M=35.5%, SD=14.9%) in terms of overall post-test scores compared to students in the control condition (M=27.1%, SD=17.2%). Further analysis, including statistical tests for significance, is forthcoming....

[[Image:graph40.jpg|500px]]

=== Explanation ===

There is evidence that the difference in post-test scores is due to increased interest leading to deeper processing of the reading practice texts.

Responses to questionnaires following each reading show the interest level of students using the REAP tutor. The questionnaires asked students to indicate on a scale from one to five their interest inr the preceding text. The distributions of post-reading interest ratings for students in the treatment and control conditions are shown in Figures 1 and 2.

[[Image:Interest_combined.jpg|700px]]

Students were also given an exit survey during their last week of practice with the tutor that asked them, among other questions, for to indicate whether they agreed with the statement, “Most of the readings were interesting.” The ratings were on a scale from one to five, with five indicating strong agreement and one indicating strong disagreement. Exit survey interest ratings by students in the treatment condition were significantly higher (p<0.05) than the ratings by students in the control condition. The mean response for students who received personalized readings was 3.18, while it was 2.65 for students in the control condition.

Further analysis of post-test scores reveals that students did learn more of the words that they actually practiced in REAP. The post-test contained 40 questions for target vocabulary words. Many of the students did not practice 40 words, so performance on practiced words alone was analyzed. Students in the treatment condition scored higher (N=16, M=50.3, SD=20.1) on questions for words seen in readings than did students in the control condition (N=19, M=32.4, SD=18.9). A two-tailed t-test for independent means verified that this result is statistically significant (t=2.719, df=33, p=0.005). The difference of scores between the two groups was 17.9% (95% CI = 4.5%, 31.3%), which corresponds to a large effect size of 0.85. This result indicates that [[personalization]] improved learning for the words that students saw in readings, which is in line with previous findings that intrinsic motivation leads to improved learning.

[[Image:post_just_practice.jpg|400px]]

However, students in the treatment condition that included [[personalization]] saw fewer words in their training sessions (N=16, M=12.0 , SD=1.13) than students in the control condition (N=19, M=16.3, SD=0.87) (t=-2.9, df=33, p=0.006). Average time on task was essentially the same for students in both conditions. Students in the treatment condition spent slightly longer on each reading. The main reason, however, for the difference in the average total number of words practiced was that students for whom the tutor provided personalized instruction saw fewer words (M=3.41, SD=0.55) per practice reading passage than students in the control condition (M=4.07, SD=0.83) (t=2.929, df=33, p=0.006).

Thus, when the tutor used [[personalization]] as a factor in the selection of readings, it chose readings that were less valuable according to other factors. Specifically, this result shows that by personalizing instruction, the tutor was not able to provide practice for as many words. Of course, the practice that it did provide was better, as is shown in the previous result that for words student did practice, [[personalization]] appeared to increase learning.

The reduced number of target words per text with personalization is a technical issue which can be avoided in a straightforward manner by increasing the size of the database of readings. With more readings, the tutor can find texts that both have ample target words and cover topics of personal interest.

[[Image:words_per_reading.jpg|400px]]

There is a possibility that the students in the treatment condition who were seeing fewer words in each reading were learning more of the words simply because they had fewer to learn per reading. To rule out this hypothesis, regression analyses (multiple linear regression) with overall post-test performance and performance for practiced words as the dependent variables. In both regression analyses, the number of target words per reading was not a significant predictor of performance. In fact, the number of target words per document was slightly positively correlated with post-test performance in both cases. This result seems to rule out the possibility that students were learning more target words in the treatment condition because they were seeing fewer words.

[[Long-term retention]] test results showed no reliable differences because of a small sample size. The test was administered to students who stayed in the ELI in the subsequent semester, which constituted only a fraction of the original sample.

=== Further Information ===

The following study addresses a different form of personalization, by which interactions with the learner (e.g., instructions, directions) are conducted using casual and direct rather than formal language:

[[Stoichiometry_Study | Studying the Learning Effect of Personalization and Worked Examples in the Solving of Stoichiometry Problems (McLaren, Koedinger & Yaron)]]

=== Annotated bibliography ===

Note: a paper on this study has been submitted to International Journal of Artificial Intelligence in Education.

[http://reap.cs.cmu.edu/Papers/heilman_topic_choice_AIED2007_poster_final.pdf Heilman, M., Juffs, A., & Eskenazi, M. (2007). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)]

Clark, R. C. and Mayer, R. E. (2003). e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer.

Cordova, D. I. & Lepper, M. R. (1996). Intrinsic Motivation and the Process of Learning: Beneficial Effects of Contextualization, Personalization, and Choice. Journal of Educational Psychology. Vol. 88,l No. 4, 715-730.

Lepper, M. (1988). Motivational Considerations in the Study of Instruction. Cognition and Instruction. 5(4), 289-309.

Heilman, M., Juffs, A., & Eskenazi, M. (To Appear). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)

File:Post just practice.jpg

2008-02-13T19:20:59Z

Gurpreet:

File:Words per reading.jpg

2008-02-13T19:16:56Z

Gurpreet:

REAP Study on Personalization of Readings by Topic (Fall 2006)

2008-02-13T19:15:46Z

Gurpreet: /* Explanation */

== REAP Study on Personalization of Readings for Increased Interest ==

=== Logistical Information ===

{| border="1"
|+
|-
| '''Contributors''' || Maxine Eskenazi, Alan Juffs, Michael Heilman, Kevyn Collins-Thompson, Lois Wilson, Jamie Callan
|-
| '''Study Start Date''' || September 11, 2006
|-
| '''Study End Date''' || November 21, 2006
|-
| '''Learnlab Courses''' || English Language Institute Reading 4 (ESL LearnLab)
|-
| '''Number of Students''' || 35
|-
| '''Total Participant Hours (est.)''' || 270
|-
| '''Data in Datashop''' || no
|}

=== Abstract ===

This paper discusses the enhancement of the REAP tutor to allow for [[personalization]] of reading materials
by topic in order to increase interest and motivation. In this work, the term “[[personalization]]” refers to the
selection of practice readings in order to match a student’s interests.

During each training session with REAP, students work through a series of readings, each of which is followed by
practice exercises for the target words in the reading. While reading a passage, students are able to access
dictionary definitions for any word in a reading either by clicking on a highlighted target word or by typing a
word into a box in the lower-left corner of the screen. The target words in the readings are also highlighted
because highlighting may increase the use of dictionary definitions, thus encouraging students to
coordinate multiple sources of information about a word’s meaning—namely, the implicit context around
words and the explicit definitions of words.

A problem discovered in past studies with REAP is that many students spend only a brief amount of time
on a reading and do not deeply process the text. Students often only read the dictionary definition for target
words rather than attempting to process the entire context around the words. Inferring the meaning of
vocabulary from context is a seemingly important strategy that is not used by such students. This behavior is likely due to a desire to perform well on post-reading practice exercises and post-test, which can be viewed as forms of extrinsic motivation. Intrinsically
motivated students who are more interested in a reading are more likely to read the entire text and to use
context to learn the meaning of unknown vocabulary. Therefore, [[personalization]] that increases intrinsic
motivation could lead to deeper processing of context and better learning of vocabulary.

{| border="1"
|+
|-
| ||'''Passive''' || '''Active''' || '''Interactive'''
|-
| '''Explicit (general)''' || Dictionary Definitions || || Practice Exercises
|-
| '''Implicit (instance)''' || Interpreting meaning in context while reading || Sentence Production (assessment) || Practice Exercises
|}

=== Glossary ===

''Intrinsic Motivation:'' Motivation to learn for learning's own sake rather than some external goal.

''Extrinsic Motivation:'' Motivation for learn in order to satisfy an external goal, such as completing a task or passing an assessment.

=== Research question ===

Do the benefits of [[personalization]] of practice readings by topics of interest outweigh the costs in a tutoring system for ESL vocabulary practice?

=== Dependent variables ===
[[Normal post-test]] scores

[[Normal post-test]] scores for practiced words only

[[Long-term retention]] test scores, same post-test but administered months later.

Evidence of [[Transfer]]: sentence production tasks for target words, correct use of words in writing assignments for other courses.

=== Independent variables ===
[[Personalization]] of readings by topics of interest. In the control condition, the tutor did not use potential personal interest as a factor in its selection of reading materials. In the treatment condition, the tutor did use interest as a factor. All other selection criteria were the same in both conditions. Time on task was also the same.

=== Hypotheses ===

Since intrinsic motivation seems to be important in language learning, the benefits of [[personalization]] will outweigh the costs.

=== Findings ===

Students in the treatment condition with [[personalization]] performed better on average (M=35.5%, SD=14.9%) in terms of overall post-test scores compared to students in the control condition (M=27.1%, SD=17.2%). Further analysis, including statistical tests for significance, is forthcoming....

[[Image:graph40.PNG|500px]]

=== Explanation ===

There is evidence that the difference in post-test scores is due to increased interest leading to deeper processing of the reading practice texts.

Responses to questionnaires following each reading show the interest level of students using the REAP tutor. The questionnaires asked students to indicate on a scale from one to five their interest inr the preceding text. The distributions of post-reading interest ratings for students in the treatment and control conditions are shown in Figures 1 and 2.

[[Image:Interest_combined.jpg|700px]]

Students were also given an exit survey during their last week of practice with the tutor that asked them, among other questions, for to indicate whether they agreed with the statement, “Most of the readings were interesting.” The ratings were on a scale from one to five, with five indicating strong agreement and one indicating strong disagreement. Exit survey interest ratings by students in the treatment condition were significantly higher (p<0.05) than the ratings by students in the control condition. The mean response for students who received personalized readings was 3.18, while it was 2.65 for students in the control condition.

Further analysis of post-test scores reveals that students did learn more of the words that they actually practiced in REAP. The post-test contained 40 questions for target vocabulary words. Many of the students did not practice 40 words, so performance on practiced words alone was analyzed. Students in the treatment condition scored higher (N=16, M=50.3, SD=20.1) on questions for words seen in readings than did students in the control condition (N=19, M=32.4, SD=18.9). A two-tailed t-test for independent means verified that this result is statistically significant (t=2.719, df=33, p=0.005). The difference of scores between the two groups was 17.9% (95% CI = 4.5%, 31.3%), which corresponds to a large effect size of 0.85. This result indicates that [[personalization]] improved learning for the words that students saw in readings, which is in line with previous findings that intrinsic motivation leads to improved learning.

[[Image:post_just_practice.jpg|400px]]

However, students in the treatment condition that included [[personalization]] saw fewer words in their training sessions (N=16, M=12.0 , SD=1.13) than students in the control condition (N=19, M=16.3, SD=0.87) (t=-2.9, df=33, p=0.006). Average time on task was essentially the same for students in both conditions. Students in the treatment condition spent slightly longer on each reading. The main reason, however, for the difference in the average total number of words practiced was that students for whom the tutor provided personalized instruction saw fewer words (M=3.41, SD=0.55) per practice reading passage than students in the control condition (M=4.07, SD=0.83) (t=2.929, df=33, p=0.006).

Thus, when the tutor used [[personalization]] as a factor in the selection of readings, it chose readings that were less valuable according to other factors. Specifically, this result shows that by personalizing instruction, the tutor was not able to provide practice for as many words. Of course, the practice that it did provide was better, as is shown in the previous result that for words student did practice, [[personalization]] appeared to increase learning.

The reduced number of target words per text with personalization is a technical issue which can be avoided in a straightforward manner by increasing the size of the database of readings. With more readings, the tutor can find texts that both have ample target words and cover topics of personal interest.

[[Image:words_per_reading.jpg|400px]]

There is a possibility that the students in the treatment condition who were seeing fewer words in each reading were learning more of the words simply because they had fewer to learn per reading. To rule out this hypothesis, regression analyses (multiple linear regression) with overall post-test performance and performance for practiced words as the dependent variables. In both regression analyses, the number of target words per reading was not a significant predictor of performance. In fact, the number of target words per document was slightly positively correlated with post-test performance in both cases. This result seems to rule out the possibility that students were learning more target words in the treatment condition because they were seeing fewer words.

[[Long-term retention]] test results showed no reliable differences because of a small sample size. The test was administered to students who stayed in the ELI in the subsequent semester, which constituted only a fraction of the original sample.

=== Further Information ===

The following study addresses a different form of personalization, by which interactions with the learner (e.g., instructions, directions) are conducted using casual and direct rather than formal language:

[[Stoichiometry_Study | Studying the Learning Effect of Personalization and Worked Examples in the Solving of Stoichiometry Problems (McLaren, Koedinger & Yaron)]]

=== Annotated bibliography ===

Note: a paper on this study has been submitted to International Journal of Artificial Intelligence in Education.

[http://reap.cs.cmu.edu/Papers/heilman_topic_choice_AIED2007_poster_final.pdf Heilman, M., Juffs, A., & Eskenazi, M. (2007). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)]

Clark, R. C. and Mayer, R. E. (2003). e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer.

Cordova, D. I. & Lepper, M. R. (1996). Intrinsic Motivation and the Process of Learning: Beneficial Effects of Contextualization, Personalization, and Choice. Journal of Educational Psychology. Vol. 88,l No. 4, 715-730.

Lepper, M. (1988). Motivational Considerations in the Study of Instruction. Cognition and Instruction. 5(4), 289-309.

Heilman, M., Juffs, A., & Eskenazi, M. (To Appear). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)

REAP Study on Personalization of Readings by Topic (Fall 2006)

2008-02-13T19:15:19Z

Gurpreet: /* Explanation */

== REAP Study on Personalization of Readings for Increased Interest ==

=== Logistical Information ===

{| border="1"
|+
|-
| '''Contributors''' || Maxine Eskenazi, Alan Juffs, Michael Heilman, Kevyn Collins-Thompson, Lois Wilson, Jamie Callan
|-
| '''Study Start Date''' || September 11, 2006
|-
| '''Study End Date''' || November 21, 2006
|-
| '''Learnlab Courses''' || English Language Institute Reading 4 (ESL LearnLab)
|-
| '''Number of Students''' || 35
|-
| '''Total Participant Hours (est.)''' || 270
|-
| '''Data in Datashop''' || no
|}

=== Abstract ===

This paper discusses the enhancement of the REAP tutor to allow for [[personalization]] of reading materials
by topic in order to increase interest and motivation. In this work, the term “[[personalization]]” refers to the
selection of practice readings in order to match a student’s interests.

During each training session with REAP, students work through a series of readings, each of which is followed by
practice exercises for the target words in the reading. While reading a passage, students are able to access
dictionary definitions for any word in a reading either by clicking on a highlighted target word or by typing a
word into a box in the lower-left corner of the screen. The target words in the readings are also highlighted
because highlighting may increase the use of dictionary definitions, thus encouraging students to
coordinate multiple sources of information about a word’s meaning—namely, the implicit context around
words and the explicit definitions of words.

A problem discovered in past studies with REAP is that many students spend only a brief amount of time
on a reading and do not deeply process the text. Students often only read the dictionary definition for target
words rather than attempting to process the entire context around the words. Inferring the meaning of
vocabulary from context is a seemingly important strategy that is not used by such students. This behavior is likely due to a desire to perform well on post-reading practice exercises and post-test, which can be viewed as forms of extrinsic motivation. Intrinsically
motivated students who are more interested in a reading are more likely to read the entire text and to use
context to learn the meaning of unknown vocabulary. Therefore, [[personalization]] that increases intrinsic
motivation could lead to deeper processing of context and better learning of vocabulary.

{| border="1"
|+
|-
| ||'''Passive''' || '''Active''' || '''Interactive'''
|-
| '''Explicit (general)''' || Dictionary Definitions || || Practice Exercises
|-
| '''Implicit (instance)''' || Interpreting meaning in context while reading || Sentence Production (assessment) || Practice Exercises
|}

=== Glossary ===

''Intrinsic Motivation:'' Motivation to learn for learning's own sake rather than some external goal.

''Extrinsic Motivation:'' Motivation for learn in order to satisfy an external goal, such as completing a task or passing an assessment.

=== Research question ===

Do the benefits of [[personalization]] of practice readings by topics of interest outweigh the costs in a tutoring system for ESL vocabulary practice?

=== Dependent variables ===
[[Normal post-test]] scores

[[Normal post-test]] scores for practiced words only

[[Long-term retention]] test scores, same post-test but administered months later.

Evidence of [[Transfer]]: sentence production tasks for target words, correct use of words in writing assignments for other courses.

=== Independent variables ===
[[Personalization]] of readings by topics of interest. In the control condition, the tutor did not use potential personal interest as a factor in its selection of reading materials. In the treatment condition, the tutor did use interest as a factor. All other selection criteria were the same in both conditions. Time on task was also the same.

=== Hypotheses ===

Since intrinsic motivation seems to be important in language learning, the benefits of [[personalization]] will outweigh the costs.

=== Findings ===

Students in the treatment condition with [[personalization]] performed better on average (M=35.5%, SD=14.9%) in terms of overall post-test scores compared to students in the control condition (M=27.1%, SD=17.2%). Further analysis, including statistical tests for significance, is forthcoming....

[[Image:graph40.PNG|500px]]

=== Explanation ===

There is evidence that the difference in post-test scores is due to increased interest leading to deeper processing of the reading practice texts.

Responses to questionnaires following each reading show the interest level of students using the REAP tutor. The questionnaires asked students to indicate on a scale from one to five their interest inr the preceding text. The distributions of post-reading interest ratings for students in the treatment and control conditions are shown in Figures 1 and 2.

[[Image:Interest_combined.PNG|700px]]

Students were also given an exit survey during their last week of practice with the tutor that asked them, among other questions, for to indicate whether they agreed with the statement, “Most of the readings were interesting.” The ratings were on a scale from one to five, with five indicating strong agreement and one indicating strong disagreement. Exit survey interest ratings by students in the treatment condition were significantly higher (p<0.05) than the ratings by students in the control condition. The mean response for students who received personalized readings was 3.18, while it was 2.65 for students in the control condition.

Further analysis of post-test scores reveals that students did learn more of the words that they actually practiced in REAP. The post-test contained 40 questions for target vocabulary words. Many of the students did not practice 40 words, so performance on practiced words alone was analyzed. Students in the treatment condition scored higher (N=16, M=50.3, SD=20.1) on questions for words seen in readings than did students in the control condition (N=19, M=32.4, SD=18.9). A two-tailed t-test for independent means verified that this result is statistically significant (t=2.719, df=33, p=0.005). The difference of scores between the two groups was 17.9% (95% CI = 4.5%, 31.3%), which corresponds to a large effect size of 0.85. This result indicates that [[personalization]] improved learning for the words that students saw in readings, which is in line with previous findings that intrinsic motivation leads to improved learning.

[[Image:post_just_practice.PNG|400px]]

However, students in the treatment condition that included [[personalization]] saw fewer words in their training sessions (N=16, M=12.0 , SD=1.13) than students in the control condition (N=19, M=16.3, SD=0.87) (t=-2.9, df=33, p=0.006). Average time on task was essentially the same for students in both conditions. Students in the treatment condition spent slightly longer on each reading. The main reason, however, for the difference in the average total number of words practiced was that students for whom the tutor provided personalized instruction saw fewer words (M=3.41, SD=0.55) per practice reading passage than students in the control condition (M=4.07, SD=0.83) (t=2.929, df=33, p=0.006).

Thus, when the tutor used [[personalization]] as a factor in the selection of readings, it chose readings that were less valuable according to other factors. Specifically, this result shows that by personalizing instruction, the tutor was not able to provide practice for as many words. Of course, the practice that it did provide was better, as is shown in the previous result that for words student did practice, [[personalization]] appeared to increase learning.

The reduced number of target words per text with personalization is a technical issue which can be avoided in a straightforward manner by increasing the size of the database of readings. With more readings, the tutor can find texts that both have ample target words and cover topics of personal interest.

[[Image:words_per_reading.jpg|400px]]

There is a possibility that the students in the treatment condition who were seeing fewer words in each reading were learning more of the words simply because they had fewer to learn per reading. To rule out this hypothesis, regression analyses (multiple linear regression) with overall post-test performance and performance for practiced words as the dependent variables. In both regression analyses, the number of target words per reading was not a significant predictor of performance. In fact, the number of target words per document was slightly positively correlated with post-test performance in both cases. This result seems to rule out the possibility that students were learning more target words in the treatment condition because they were seeing fewer words.

[[Long-term retention]] test results showed no reliable differences because of a small sample size. The test was administered to students who stayed in the ELI in the subsequent semester, which constituted only a fraction of the original sample.

=== Further Information ===

The following study addresses a different form of personalization, by which interactions with the learner (e.g., instructions, directions) are conducted using casual and direct rather than formal language:

[[Stoichiometry_Study | Studying the Learning Effect of Personalization and Worked Examples in the Solving of Stoichiometry Problems (McLaren, Koedinger & Yaron)]]

=== Annotated bibliography ===

Note: a paper on this study has been submitted to International Journal of Artificial Intelligence in Education.

[http://reap.cs.cmu.edu/Papers/heilman_topic_choice_AIED2007_poster_final.pdf Heilman, M., Juffs, A., & Eskenazi, M. (2007). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)]

Clark, R. C. and Mayer, R. E. (2003). e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer.

Cordova, D. I. & Lepper, M. R. (1996). Intrinsic Motivation and the Process of Learning: Beneficial Effects of Contextualization, Personalization, and Choice. Journal of Educational Psychology. Vol. 88,l No. 4, 715-730.

Lepper, M. (1988). Motivational Considerations in the Study of Instruction. Cognition and Instruction. 5(4), 289-309.

Heilman, M., Juffs, A., & Eskenazi, M. (To Appear). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)

File:Summary Table.jpg

2008-02-13T19:12:19Z

Gurpreet:

Visual Representations in Science Learning

2008-02-13T19:11:21Z

Gurpreet: /* Summary Table */

== Visual Representations in Science Learning ==
Jodi Davenport

===Summary Table===
Total Studies: 7 
Total Participants: 1652 
Total Participant hours: 7461 

Additional Contributors
* David Yaron
* David Klahr
* Ken Koedinger
* Michael Karabinos
* Gaea Leinhardt
* Jim Greeno
* Katie McEldoon

[[Image:Summary Table.jpg]]

=== Abstract ===
Visual representations, in the forms of diagrams, notation (e.g., equations), graphs and tables are fundamental tools in science instruction and practice. Whether diagrams or notational systems are helpful aids to problem solving depends critically on the content of the visual representation and how learners are able to process the information they contain. Expert/novice studies have demonstrated that different levels of experience result in differential processing of the same stimuli. However, it is not known how students are able to refine initially shallow understandings into meaningful chemical concepts or how the [[coordination]] of multiple representations helps with this process.

The current project seeks to determine when and how the use of multiple representations during instruction and problem solving will lead to [[robust learning]]. To date, 7 studies (4 completed, 3 ongoing) have explored science learning at the two levels (micro and macro) of the theoretical framework.

'''Microscopic Level: Identifying [[knowledge components]] and developing assessments'''
In order to assess robust learning, we conducted studies and collaborated with Chemistry and Education faculty (David Yaron, Gaea Leinhardt & Jim Greeno) to identify key knowledge components of equilibrium and acid base chemistry. For instance, a verbal protocol study has demonstrated that experts and novices differ in their ability to invoke relevant knowledge components in contexts involving different representations (e.g., chemical equations, graphs, and diagrams). We continue to create and revise new forms of assessments of knowledge [[transfer]] that identify which correct and incorrect knowledge components students have and apply as they learn chemistry.

'''Macroscopic Level: Testing general learning principles'''
At the macroscopic level, ''in vivo'' studies test general learning principles. Studies have investigated whether the use of molecular-level diagrams increases [[robust learning]] as measured by [[transfer]] performance and have manipulated conditions to determine what type of instructional prompts will promote active processing. Early studies failed to find a learning advantage for molecular level diagrams and ongoing studies seek to determine what conditions may be required to produce a benefit of multiple representations during instruction.

=== Glossary ===
Visual representations: [[External representations]] that are used in instruction and problem solving such as diagrams, graphs, and equations

=== Hypothesis ===
Chemical equilibrium is a difficult topic for students to learn as it involves learning a large set of [[knowledge components]] and applying these components flexibly in a variety of situations. As different representations make salient different key features of knowledge components, we hypothesize that instruction that requires the [[coordination]] of multiple representations will lead to more [[robust learning]] as measured by [[transfer]] tests such as conceptual multiple choice questions and open-ended inference questions.

=== Research Questions ===
Our project seeks to identify the [[knowledge components]] of equilibrium and acid base chemistry and determine when and how [[coordination]] of different types of representations lead to the acquisition of correct knowledge components and [[robust learning]].

*What are the knowledge components of equilibrium and acid/base chemistry? (CMU, lab study #1, 2006; Chemistry working group)
*How do experts and novices differ in equilibrium problem solving with multiple representations? (CMU, lab study #1, 2006)
*Does the presence of molecular level diagrams enhance robust learning of acid/base chemistry? (UBC, in vivo study #1, 2006; CMU, in vivo study #2, 2006)
*Do molecular level diagrams enhance self explanation leading to robust learning in a tutorial on acids and bases? (CMU, lab study #2, 2006)
*Do labelled diagrams enhance robust learning as measured by transfer performance? (UBC, in vivo study #3, 2007; CMU, in vivo study #5, 2007)
*Do virtual lab activities (in which multiple representations must be coordinated) enhance performance on interactive problem solving and transfer questions? (UBC, in vivo study #3, 2007)
*Does instruction with multiple representations including molecular level diagrams and graphs enhance the acquisition of equilibrium knowledge components leading to robust learning? (CMU, in vivo study #4, 2007)

=== Background and Significance ===
Robust learning in chemistry is more than learning a set of facts or procedures. Instead, many pieces of knowledge must be acquired and flexibly coordinated in order to make predictions, inferences and analyses of chemical systems. We chose to focus on chemical equilibrium systems as the knowledge is important for understanding processes in chemistry, biology and engineering and the domain is notoriously difficult for students (e.g., Banerjee, 1991; Van Driel et al. 1999).

As the knowledge components of chemical equilibrium are not necessarily explicit in textbooks or instructional materials, we have formed a Chemistry working group with Chemistry, Education and Learning Science faculty to identify core pieces of understanding in this domain. In addition, verbal protocol analyses help us understand student how students process instruction and solve chemistry problems.

Our hypothesis is that instruction and practice that includes a variety of representations will lead to robust learning. Multimedia instruction is widely believed to help chemistry learning by providing a bridge between the mathematical procedures and core chemistry concepts. For instance, chemistry education researchers have suggested that the ability to transform between representations leads to increased success in problem solving (Bodner & Domin, 2000) and that including diagrams during classroom instruction leads to improved conceptual understanding (Ardac & Akaygun, 2004; Bunce & Gabel, 2002; Noh & Scharmann, 1997; Sanger & Badger, 2001; Williamson & Abraham, 1995;see Kozma & Russell, 2005 for a review). Though the results of classroom studies are promising, the interventions often lasted many class periods and the content in the experimental and control classes was not tightly controlled, so it is unclear exactly how specific representations influenced learning. Further, assessments that demonstrated greater conceptual understanding did not always show increased problem solving performance.

In a number of laboratory studies, Mayer (Clark & Mayer, 2003), Ainsworth & Loizou (2003) and others have found that instructional materials that include both diagrams and text provide learning benefits over materials that only include text. Will this same learning advantage extend to classroom-based instruction? Studies in chemistry education research have suggested that molecular level diagrams may promote deeper understanding that text or equations. However, these classroom-based studies lacked rigorous controls and assessments of robust learning. The current project investigates if and when the presence of visual representations in addition to text promotes deep conceptual understanding in chemistry.

=== Independent Variables ===
The studies in the project manipulate the presence of visual representations in chemistry instruction and problem solving.

For instance, in the (CMU, 2006) Lab Study #1, experts and novices solved problems in diffferent representational formats. The traditional format, found in many textbooks, involves a chemical equation and text-based setup. The diagram format requires solvers to additionally integrate pictorial information. 
[[Image:trad_diag.jpg]] 

In studies testing the role of molecular-level representations in chemistry instruction (Lab Study #2, ''in vivo'' Studies #1, #2, #3 and #5), text is identical in both conditions and the diagram condition supplements the text with pictures. 

[[Image:mole_pic.jpg]]

=== Dependent Variables ===
Dependent variables include improvement from pre to post test on conceptual multiple choice questions, performance on scaffolded problem solving with tutors and performance on open-ended [[transfer]] questions.

''[[Transfer]] Questions'' 
Our studies use a range of transfer assessments
*Conceptual Multiple Choice
**E.g. You have 5 weak acids in your laboratory and want to create a series of buffer solutions. If the target pH is 4.3 which weak acid you would use to create your buffer, and would you use more weak base, weak acid or the same amount of each?
:::Weak acid A pKa = 7.35
:::Weak acid B pKa = 7.15
:::Weak acid C pKa = 6
:::Weak acid D pKa = 4.3
:::Weak acid E pKa = 6.5
*Open-ended Transfer Questions
** E.g., "How could you make a solution of lemon juice that is more acidic than a solution of hydrocholoric acid?"
*Scaffolded Problem Solving with tutors
*Virtual Laboratory Problems

''[[Normal post-test]]'' Assessments 
Include:
*Definitions: Defining terms such as acid/base
*True/False questions

=== Findings ===

====Expert/Novice Equilibrium Problem Solving: Lab study #1 ====
N = 16 (CMU, 2006) Lab Study #1

The two aims of this study were 1) to create a taxonomy of knowledge components for equilibrium chemistry and 2) to determine whether problem representation led to differences in application of knowledge components.

A knowledge decomposition was carried out on the transcribed protocols and a taxonomy of equilibrium knowledge components has been created.

=====Findings Summary=====
To assess whether problem representation influenced the application of knowledge components, experts and novices solved equilibrium problems in different contexts while talking aloud. While experts were equally able to retrieve the correct [[knowledge component]] (in this case that the equilibrium constant, K, was required for problem solving), novices were able to retrieve the correct knowledge component when solving a traditional problem, but were less successful on problems using molecular-level diagrams. There was a significant main effect of expertise, F(1,13) = 9.15, p = .01, and an interaction between expertise and problem type, F (1, 13) = 5.205, p = .04. 

[[Image:K_exnov.jpg]] 

====Learning with molecular level diagrams: ''In vivo'' studies #1 and #2====
N = 42 (UBC, 2006) ''in vivo'' Study #1 
N = 89 (CMU, 2006) ''in vivo'' Study #2

The aim of these studies was to determine whether molecular level diagrams lead to increased performance on tranfer questions after reading a tutorial that either contained Text+Diagram, or Text Only.

Results (UBC, 2006) ''in vivo'' Study #1
*An ANCOVA was run on posttest scores with format (Diagram+Text vs. Text Only) as a between-subjects variable and pretest scores as a covariate. No significant effect of condition was found, F(1, 39) = .025, p = .88. Posttest scores were similar regardless of whether students were in the diagrams (M =.72) or text-only (M = .72) conditions.

Results (CMU, 2006) ''in vivo'' Study #2
*A mixed 2x2 ANOVA was carried out with format (Diagram+Text vs. Text-only) as a between-subjects variable, time of test (Pre vs. Posttest) as a within-subjects variable and multiple-choice accuracy as the dependent variable. Pretest to posttest gains were significant, F(1, 87) = 94.4, p < .001, but there was no effect of format, and no interaction between test-time and format. See table below.

::[[image:Earli_tables.gif]]

=====Findings summary=====
To date, results suggest no advantage for the addition of diagrams to text. Specifically, acid/base chemistry tutorials that included molecular-level diagrams did not produce enhanced learning compared with text-only versions of the same tutorials. Future studies will investigate whether labelled diagrams will be more likely to promote [[integration]] of textual material with visual representations, leading to more robust learning of chemistry concepts.

====Molecular level diagrams and self explanation: Lab study #2====
N = 22 (CMU, 2006) Lab Study #2

As no benefits were found from the ''in vivo'' studies, we conducted a lab study to determine whether there were any processing differences in the Text+Diagram and Diagram Only conditions. Participants were instructed to self explain as they read through a tutorial on acid/base chemistry.

=====Findings Summary=====
* Students made substantial learning gains from pre to post-test, p < .001. However, no significant main effect of condition was found for either the multiple choice test or performance on the definition questions F(1,20) < 1. Further, no main effect of condition was found for performance on transfer questions.
:[[Image:CMUabtut.jpg]]

* Transcribed protocols were analysed for time on task and number of words. There were no significant differences between conditions.
* Verbal protocols were coded for Self Explanations (Noticing Coherence, Elaborations or Principle Based) and Monitoring statements (Positive or Negative). No significant differences in number of self explanations were found between conditions.
* Correlations: Significant correlations were found between the number of self explanations (SE) and transfer performance. In the Text Only condition, SE total was positively correlated with transfer performance (.642), however in the Diagrams+Text condition, SE total was negatively correlated with transfer performance (-.606). These results suggest that the diagrams in this particular study may have generated less germane processing. An ongoing analysis of this data will investigate whether SEs in the Diagram+Text condition were more shallow than in the Text Only condition.

====Learning from labelled molecular level diagrams and Virtual Lab activities: ''In vivo'' study #3, UBC, 2007====
N = 1139 (UBC, 2007) ''in vivo'' Study #3

This ongoing study is an extension of ''in vivo'' Studies #1 and #2. The aims of this study are 1) to determine whether labelled diagrams would lead to greater learning advantages and 2) To determine whether Virtual Lab activities enhance problem-solving performance with scaffolded, interactive tutors. The lack of effect of diagrams in the earlier studies may have been due to the lack of labels with the diagrams. Prior work may have confounded the study of labels with diagrams and it is possible that only labelled diagrams promote the active processing required for learning gains from the [[coordination]] of representations.

The 2x2 design varies format (Diagrams + Text vs. Text Only) and the order of Virtual Lab and problem-solving tutors (Virtual Lab first vs. Virtual Lab second.)

====The influence of multiple representations on knowledge component acquisition in chemical equilibrium: ''In vivo'' study #4====
N = 172 (CMU, 2007) ''in vivo'' Study #4

Our knowledge decomposition of chemical equilibrium (informed by Lab Studies #1 and #2) as well as discussions with our Chemistry working group have revealed that the key knowledge component of "progress of reaction" is left implicit in many types of traditional instruction. For this ongoing study two sets of online lectures were created by Prof. Yaron to determine if instruction that uses multiple representations to convey the notion of progress of reaction would lead to more robust learning of chemistry concepts.

[[Transfer]] measures of open-ended responses and conceptual multiple choice questions were collected, and measures of [[long term retention]] on a follow-up quiz, in class "clicker" responses and exam scores will be analyzed.

====Molecular level diagrams and robust learning of acid base buffer concepts: ''In vivo'' study #5====
N = 172 (CMU, 2007) ''in vivo'' Study #5

This ongoing study is an extension of ''in vivo'' studies #1 and #2 that manipulate the presence of molecular level diagrams in a Chemistry tutorial. In addition to collecting pretest and posttest [[transfer]] measures, we are also collecting transfer performance on each page of the tutorial to determine which knowledge components are accessed in the Diagram+Text versus the Diagram Condition.

=== Explanation ===
The knowledge decomposition of chemical equilibrium has revealed a number of correct knowledge components to be acquired by students and incorrect knowledge components to be avoided. This taxonomy has been used to create assessments used in our studies.

Our expert-novice protocol study revealed that experts are more likely to invoke a relevent [[knowledge component]] across different problem types than novice solvers. This result suggests that many students maintain a shallow understanding of chemical systems even after completing a year of college-level chemistry. Current studies are addressing whether instruction that ties the core concept of progress of reaction (identified through the expert/novice studies) to multiple representations will lead to more [[robust learning]].

Our studies testing learning benefits for visual diagrams during instruction suggest the large effects of diagrams commonly found in laboratory studies of topics such as bicycle pumps, lightning formation and disc brakes may be difficult to replicate in educational settings. Active and intentional coordination of representations may be required if diagrams are to increase learning and the mere presence of diagrams does not guarantee this type of active processing. Current studies seek to determine whether labelled diagrams will enhance the coordination of text and diagrams via [[sense making]]. Further, prior work has not closely mapped the features of to-be-learned material to the information contained in diagrams and text. In our ongoing studies, we are investigating which pieces of information are contained in text, which are contained in the diagrams and which features students are able to extract when they are presented with multiple representations.

=== Publications and Presentations ===

Davenport, J.L., McEldoon, K. & Klahr, K. (2007). Depicting invisible processes: The influence of molecular-level diagrams in Chemistry instruction. To be presented at the 29th Annual meeting of the Cognitive Science Society. August 2007. [http://www.learnlab.org/uploads/mypslc/publications/cogsci07davenport.pdf download]

Davenport, J.L., Yaron, D., Karabinos, M., Klahr, K. & Koedinger, K. (2007). Chemical equilibrium: an evaluation of a new type of instruction. To be presented at the Gordon Conference for Chemistry Education Research and Practice. June 2007. [http://www.learnlab.org/uploads/mypslc/publications/davenport_gordon07.pdf download]

Davenport, J.L., Klahr, D. & Koedinger (2007). The influence of diagrams on chemistry learning. Paper accepted for the 12th Biennial Conference of the European Association for Research on Learning and Instruction. August 2007. [http://www.learnlab.org/uploads/mypslc/publications/davenportearli07.pdf download]

Davenport, J.L., Klahr, D. & Koedinger (2006). The influence of external representations on chemistry problem solving. Poster presented at the Forty-seventh Annual Meeting of the Psychonomic Society in Houston, Texas. November 2006. [http://www.learnlab.org/uploads/mypslc/publications/davenport06.pdf download]

Yaron, D., Karabinos, M., Davenport, J. & Leinhardt, G. Virtual labs and scenario-based activities for introductory chemistry. American Chemical Society - Penn-Ohio Regional Meeting, Theil College, Greenville, PA, October 2006.

Yaron, D., Karabinos, M., Davenport, J. & Leinhardt, G. Virtual lab activities for introductory chemistry labs, American Chemical Society Annual Meeting, San Francisco, September 2006.

Yaron, D., Karabinos, M., Davenport, J. & Leinhardt, G. Virtual labs activities for introductory chemistry. Biennial Conference on Chemical Education, Purdue University, West Layefette In, July 2006.

Yaron, D., Karabinos, M., Davenport, J., Cuadros, J., Rehm, E., McCue, W., Dennis, D., Leinhardt, G. and Evans, K. The ChemCollective Virtual Lab and Other Online Materials. Presented at Duke University, Durham, NC, November 2005.

=== Further Information ===

[[Category:Study]]

Effect of adding simple worked examples to problem-solving in algebra learning

2008-02-13T19:10:32Z

Gurpreet: /* Glossary */

''Lisa Anthony, Jie Yang, Kenneth R. Koedinger''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Lisa Anthony, Jie Yang, & Ken Koedinger
|-
| '''Other Contributers''' || n/a
|-
| '''Study Start Date''' || December 4, 2006
|-
| '''Study End Date''' || December 20, 2006
|-
| '''LearnLab Site''' || Central Westmoreland Career & Technology Center (CWCTC)
|-
| '''LearnLab Course''' || Algebra
|-
| '''Number of Students''' || 38
|-
| '''Total Participant Hours''' || 114
|-
| '''DataShop''' || To be completed ASAP
|}

=== Abstract ===
This ''in vivo'' experiment compared differences in learning that occur when students problem solve vs when they problem solve aided by worked [[example]]s. Students worked in the standard Cognitive Tutor Algebra lesson on 2-step problems. Those in the worked examples condition copied the worked example given to them using the solver's interface the first time they saw a particular problem type (''i.e.'', ax+b=c or a/x=c); following that, an analogous example would appear each time the students solve a similar problem.

The hypothesis of this study was that students who were given the worked examples would experience improved learning in both normal learning and in terms of the [[robust learning]] measures of [[transfer]] and [[accelerated future learning]]. Copying the problem the first time the students encountered a particular problem type acts as additional scaffolding for students to solve the problems.

Results are forthcoming.

=== Glossary ===
Forthcoming, but will probably include
* Sample worked-out-example:
[[Image:lanthony-example-unit9.jpg]]

=== Research question ===
Is robust learning affected by the addition of scaffolded worked examples to the problem-solving process?

=== Background & Significance ===
...Worked examples studies undergone at PSLC and beyond...

See VanLehn's paper on students using examples -- copying vs. as feedback ...
Lefevre & Dicksen ... (1986). Cognition and Instruction.

See Koedinger & Aleven's Assistance Dilemma explanation ...

=== Independent Variables ===
One independent variable was used:
* Inclusion of worked example: present or not present.

=== Hypothesis ===
The inclusion of worked examples during the problem-solving process will have benefits for learning by virtue of the scaffolding provided by having the students copy the example the first time they see a particular problem type.

=== Dependent variables ===
* ''Near [[transfer]], immediate'': Students were given a 15-minute post-test after their sessions with the computer tutor had concluded.

* ''Near transfer, [[retention]]'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there was any difference in performance at the start of that lesson.

* ''Far transfer'': Far transfer items such as 3-step problems and literal equations were included on the immediate post-test.

* ''[[Accelerated future learning]]'': We intend to analyze the log data from the students' Cognitive Tutor usage in the equation solving unit that followed the 2-step problems, to determine if there were learning curve differences during training.

=== Findings ===
Final findings in progress.

=== Explanation ===
This study is part of the [[Coordinative Learning]] cluster and addresses the examples and explanation sub-group.

The students were given examples throughout their use of the tutor. On the first instance of a particular problem type, students were asked to copy out a worked example; on subsequent instances, examples remained on the screen while students solved analogous problems.

=== Descendants ===

None.

=== Annotated Bibliography ===

Analysis and write-up in progress.

=== Further Information ===
Connected to [[Lab study proof-of-concept for handwriting vs typing input for learning algebra equation-solving]] in the [[Refinement and Fluency]] cluster.

=====Plans for June 2007-December 2007=====

* Complete transition of log data to DataShop.
* Analyze data to determine effect of including examples on pre to post test gains and/or learning curves.
* Write up results for publication in a learning science conference.
* Lab study comparing alternative methods of delivering and presenting worked examples is a possible side avenue for the parent project of this study ([[Handwriting Algebra Tutor]]).

File:Lanthony-example-unit9.jpg

2008-02-13T19:10:13Z

Gurpreet:

File:Rectangle-area learning curve.jpg

2008-02-13T19:07:17Z

Gurpreet:

Using learning curves to optimize problem assignment

2008-02-13T19:04:33Z

Gurpreet: /* Background and significance */

=== Abstract ===
This study examined the effectiveness of an educational data mining method – Learning Factors Analysis (LFA) – on improving the learning efficiency in the Cognitive Tutor curriculum. LFA uses a statistical model to predict how students perform in each practice of a knowledge component (KC), and identifies over-practiced or under-practiced KCs. By using the LFA findings on the Cognitive Tutor geometry curriculum, we optimized the curriculum with the goal of improving student learning efficiency. With a control group design, we analyzed the learning performance and the learning time of high school students participating in the Optimized Cognitive Tutor geometry curriculum. Results were compared to students participating in the traditional Cognitive Tutor geometry curriculum. Analyses indicated that students in the optimized condition saved a significant amount of time in the optimized curriculum units, compared with the time spent by the control group. There was no significant difference in the learning performance of the two groups in either an immediate post test or a two-week-later retention test. Findings support the use of this data mining technique to improve learning efficiency with other computer-tutor-based curricula.

=== Glossary ===
Data mining, intelligent tutoring systems, learning efficiency

=== Research question ===
How can we achieve a higher learning efficiency by reducing unnecessary over-practice in the intelligent tutoring settings?

=== Background and significance ===
Much intelligent tutoring system (ITS) research has been focused on designing new features to improve learning gains measured by the difference between pre and post test scores. However, learning time is another principal measure in the summative evaluation of an ITS. Intelligent tutors contribute more to education when they accelerate learning [9]. Bloom’s “Two Sigma” effect of a model human tutor [4] has been one of the ultimate goals for most intelligent tutors to achieve. So should be the “Accelerated Learning” effect shown by SHERLOCK’s offering four-year’s trouble shooting experience in the space of seven days of practice [12].

Cognitive Tutors are an ITS based on cognitive psychology results [11]. Students spend about 40% of their class time using the software. The software is built on cognitive models, which represent the knowledge a student might possess about a given subject. The software assesses students’ knowledge step by step and presents curricula tailored to individual skill levels [11]. According to Carnegie Learning Inc., by 2006, Cognitive Tutors have been widely used in over 1300 school districts in the U.S. by over 475,000 secondary school students. With such a large user base, the learning efficiency with the Tutor is of great importance. If every student saves four hours of learning over one year, nearly two million hours will be saved. To ensure adequate yearly progress, many schools are calling for an increase in instructional time. However, the reality is that students have a limited amount of total learning time, and teachers have limited amount of instructional time. Saving one hour of learning time can be better than increasing one hour of instructional time because it does not increase students’ or teachers’ work load. Moreover, if these saved hours are
spent on learning other time-consuming subjects, they can improve the learning gains in those subjects.

Educational data mining is an emerging area, which provides many potential insights that may improve education theory and learning outcomes. Much educational data mining to date has stopped at the point of yielding new insights, but has not yet come full circle to show how such insights can yield a better intelligent tutoring system (ITS) that can improve student learning [2, 3].
Learning Factors Analysis (LFA) [6, 5] is a data-mining method for evaluating cognitive models and analyzing student-tutor log data. Combining a statistical model [10], human expertise and a combinatorial search, LFA is able to measure the difficulty and the learning rates of knowledge components (KC), predict student performance in each KC practice, identify over-practiced or under-practiced KCs, and discover “hidden” KCs interpretable to humans.
[[Image:LFA_Formula.jpg]]
Pijt is the probability of getting a step in a tutoring question right by the ith student’s tth opportunity to practice the jth KC. The model says that the log odds of Pijt is proportional to the overall “smarts” of that student (θi) plus the “easiness” of that KC (βj) plus the amount gained (γj) for each practice opportunity. With this model, we can show the learning growth of students at any current or past moment.

By applying LFA to the student log data from the Area unit of the 1997 Geometry Cognitive Tutor, we found two interesting phenomena. On the one hand, some easy (i.e. high βj) KCs with low learning rates (i.e. low γj) are practiced many times. Few improvements can be made in the later stages of those practices. KC rectangle-area is an example. This KC characterizes the skill of finding the area of a rectangle, given the base and height. As shown in Figure 1, students have an initial error rate around 12%. After 18 times of practice, the error rate reduces to only 8%. The average number of practices per student is 10. Many practices spent on an easy skill are not a good use of student time. Reducing the amount of practice for this skill may save student time without compromising their performance. Other over-practiced KCs include square-area, and parallelogram-area. On the other hand, some difficult (i.e. low βj) KCs with high learning rates (i.e. high γj) do not receive enough practice. Trapezoid-area is such an example in the unit. But students received up to a maximum of 6 practices. Its initial error rate is 76%. By the end of the 6th practice the error rate remains as high as 40%, far from the level of mastery. More practice on this KC is needed for students to reach mastery. Other under-practiced KCs include pentagon-area and triangle-area.

[[Image: Rectangle-area_learning_curve.jpg]]

[[Image: Pentagon-area_learning_curve.jpg]]

Having students practice less than needed is clearly undesirable in the curriculum. Is over practice necessary? The old idiom “practice makes perfect” suggests that the more practice we do on a skill, the better we can apply the skill. Many teachers believe that giving students more practice problems is beneficial and “would like to have the students work on more practice problems”, even when “[students] were not making any mistakes and were progressing through the tutor quickly”[7].
We believe that if the teachers want more problems for their students to practice unmastered KCs or useful KCs not covered by the curriculum, more practice is necessary. To support KC long-term retention, more practice is necessary but needs to be spread on an optimal schedule [1, 14]. In the rectangle-area example, where all the practice for this KC is allocated in a short period, more practice becomes over practice, which is unnecessary after the KC is mastered.

=== Dependent variables ===

[[Normal post-test]] -- Shortly after finishing the 6th unit, the students took the post test.

[[Long-term retention]] -- In two weeks after each student finished the post test, we gave each student a retention test.

=== Independent variables ===
The independent variable was whether students used the version of the tutor with the optimized parameter settings or whether they used the version with the original parameter settings (an [[ecological control group]]).

=== Hypothesis ===
Over-practice is not necessary for short term retention as well as long term retention.

Reducing over-practice can improve learning efficiency.

=== Findings ===
The optimized group learned as much as the control group but in less time. The two groups have similar scores in both the pre test and the post test. The amount of learning gain in both groups is approximately 5 points. To further examine the treatment effect, we ran an ANCOVA on the post test scores, with condition as a between subject factor, the pretest scores as a covariate, and an interaction term between the pretest scores and the condition. The post test scores are significantly higher than the pretest, p < .01, suggesting that the curriculum overall is effective. Meanwhile neither the condition nor the interaction are significant, p = 0.772, and p = 0.56 respectively. As shown in the Figure 2 (right), we found no significant difference in the retention test scores (p = 0.602, two tailed). The results from the post test and the retention tests suggest that there is no significant difference between the two groups on either of the two tests. Thus, over practice does not lead to a significantly higher learning gain.

[[Image: Pre_post_test.jpg]]

[[Image: Retention_test.jpg]]

The actual learning time in each unit matches our hypotheses. As shown in Table 2, the students in the optimized condition spent less time than the students in the control condition in all the units except in the circle unit. The optimized group saved the most amount of time, 14 minutes, in unit 1 with marginal significance p = .19; 5 minutes in unit 2, p = .01, and 1.92, 0.49, 0.28 minutes in unit 3, 4, and 5 respectively. In unit 6, where we lowered P(L0), the optimized group spent 0.3 more minutes. Notice the percentage of the time saved in each unit. The students saved 30% of tutoring time in unit 2 Parallelogram, and 14% in unit 1 Square. In total students in the optimized condition saved around 22 minutes, an 12% reduction in the total tutoring time.

[[Image: Time_saving_table.jpg]]

=== Explanation ===

[[Knowledge component hypothesis]]
The over practice is the amount of practice done after students have mastered the skills under measurement. Once the skills reach the mastery level, any more practice on them may only leads to very little learning gain. Those gain are not even retained over the longer term. By removing those over practice, students can learn as much but save more time.

=== Descendants ===

=== Annotated bibliography ===

[1] J. R. Anderson, J. M. Fincham and S. A. Douglass, Practice and retention: A unifying analysis Journal of Experimental Psychology: Learning, Memory, and Cognition, 25 (1999), pp. 1120-1136.

[2] T. Barnes, The Q-matrix Method: Mining Student Response Data for Knowledge, American Association for Artificial Intelligence 2005 Educational Data Mining Workshop, 2005.

[3] J. E. Beck and B. P. Woolf, Evaluating tutorial actions with a user model, User Modeling and User Adapted Interaction, 2007.

[4] B. S. Bloom, The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring, Educational Researcher, 13 (1984), pp. 4-16

[5] H. Cen, K. Koedinger and B. Junker, Automating cognitive model improvement by a* search and logistic regression, American Association for Artificial Intelligence 2005 Workshop on Educational Datamining, 2005.

[6] H. Cen, K. Koedinger and B. Junker, Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement, 8th International Conference on Intelligent Tutoring Systems, 2006.

[7] J. Cital, A business email report to Carnegie Learning Inc, 2006, pp. Email.

[8] A. T. Corbett and J. R. Anderson, Knowledge tracing: Modeling the acquisition of procedural knowledge, User Modeling and User-Adapted Interaction, 1995, pp. 253-278.

[9] A. T. Corbett, K. Koedinger and J. R. Anderson, Intelligent Tutorins Systems, in M. G. Helander, T. K. Landauer and P. Prabhu, eds., Handbook of Human-Computer Interaction, Elsevier Science, Amsterdam, The Netherlands, 1997.

[10] K. Draney, P. Pirolli and M. Wilson, A Measurement Model for a Complex Cognitive Skill, Cognitively Diagnostic Assessment, Erlbaum, Hillsdale, NJ 1995.

[11] K. R. Koedinger and A. T. Corbett, Cognitive Tutors: Technology Bringing Learning Science to the Classroom, in K. Sawyer, ed., The Cambridge Handbook of the Learning Sciences, Cambridge University Press., 2006, pp. 61-78.

[12] A. Lesgold, S. Lajoie, M. Bunzo and G. Eggan, Sherlock: A Coached Practice Environment for an Electronics Troubleshooting Job, in J. H. Larkin and R. W. Chabay, eds., Computer-assisted instruction and intelligent tutoring systems: shared goals and complementary approaches, Lawrence Erlbaum Associates, 1988.

[13] A. Mitrovic, M. Mayo, P. Suraweera and B. Martin, Constraint-based tutors: a success story, 14th Int. Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems IEA/AIE-2001, Springer-Verlag Berlin Heidelberg, Budapest, 2001.

[14] P. I. Pavlik and J. R. Anderson, Practice and Forgetting Effects on Vocabulary Memory: An Activation Based Model of the Spacing Effect Cognitive Science, 29 (2005), pp. 559-586.

[15] A. Rafferty and M. Yudelson, Applying LFA for Building Stereotypical Student Models, PSLC Summer School 2006 Projects, Pittsbugh Science of Learning Center, Pittsburgh PA, 2006.

File:Pentagon-area learning curve.jpg

2008-02-13T19:02:55Z

Gurpreet:

File:Int writing tutor hyp1.gif

2008-02-05T04:12:17Z

Gurpreet: uploaded a new version of "Image:Int writing tutor hyp1.gif"

Probability of Correct Response given time for no transfer and negative transfer

File:Int writing tutor hyp2.gif

2008-02-05T04:12:02Z

Gurpreet: uploaded a new version of "Image:Int writing tutor hyp2.gif"

Probability of correct response given time for no transfer (PcNo), negative transfer(Pc), and probability of incorrect(PI)

File:Hint.jpg

2008-02-05T04:11:46Z

Gurpreet: uploaded a new version of "Image:Hint.jpg"

A typical hint in the Geometry Cognitive Tutor

File:Geometry Cognitive Tutor.jpg

2008-02-05T04:11:29Z

Gurpreet: uploaded a new version of "Image:Geometry Cognitive Tutor.jpg"

File:The Help-tutor.jpg

2008-02-05T04:11:13Z

Gurpreet: uploaded a new version of "Image:The Help-tutor.jpg"

File:The Help Seeking Model.jpg

2008-02-05T04:10:59Z

Gurpreet: uploaded a new version of "Image:The Help Seeking Model.jpg"

File:The Self-Assessment Tutor.jpg

2008-02-05T04:10:44Z

Gurpreet: uploaded a new version of "Image:The Self-Assessment Tutor.jpg"

File:Explicit help-seeking instruction.jpg

2008-02-05T04:10:20Z

Gurpreet: uploaded a new version of "Image:Explicit help-seeking instruction.jpg"

File:Embedded hints.jpg

2008-02-05T04:10:06Z

Gurpreet: uploaded a new version of "Image:Embedded hints.jpg"

File:AFL results.JPG

2008-02-05T04:01:20Z

Gurpreet: uploaded a new version of "Image:AFL results.JPG"

File:FTR results.JPG

2008-02-05T04:01:10Z

Gurpreet: uploaded a new version of "Image:FTR results.JPG"

File:Butcher TableScreenShot.jpg

2008-02-05T04:00:56Z

Gurpreet: uploaded a new version of "Image:Butcher TableScreenShot.jpg"

Screen shot of Noncontiguous tutor, with verbal explanations only.

File:Butcher UnsolvableExplanations.jpg

2008-02-05T04:00:42Z

Gurpreet: uploaded a new version of "Image:Butcher UnsolvableExplanations.jpg"

Last updated January 2007.

File:Butcher FalseExplanations.jpg

2008-02-05T04:00:28Z

Gurpreet: uploaded a new version of "Image:Butcher FalseExplanations.jpg"

Last Updated January 2007

File:Butcher DiagramScreenShot.jpg

2008-02-05T04:00:17Z

Gurpreet: uploaded a new version of "Image:Butcher DiagramScreenShot.jpg"

File:Butcher TableScreenShot2.jpg

2008-02-05T04:00:06Z

Gurpreet: uploaded a new version of "Image:Butcher TableScreenShot2.jpg"

File:Mc res.jpg

2008-02-05T03:59:07Z

Gurpreet: uploaded a new version of "Image:Mc res.jpg"

File:Def res.jpg

2008-02-05T03:58:50Z

Gurpreet: uploaded a new version of "Image:Def res.jpg"

File:Trans res.jpg

2008-02-05T03:58:38Z

Gurpreet: uploaded a new version of "Image:Trans res.jpg"

File:K exnov.jpg

2008-02-05T03:58:05Z

Gurpreet: uploaded a new version of "Image:K exnov.jpg"

File:Trad diag.jpg

2008-02-05T03:57:50Z

Gurpreet: uploaded a new version of "Image:Trad diag.jpg"

File:Mole pic.jpg

2008-02-05T03:57:36Z

Gurpreet: uploaded a new version of "Image:Mole pic.jpg"

File:CSE screen shot.jpg

2008-02-05T03:57:17Z

Gurpreet: uploaded a new version of "Image:CSE screen shot.jpg"