Difference between revisions of "Craig observing"

From LearnLab
Jump to: navigation, search
(Summary)
(Hypothesis)
 
(51 intermediate revisions by 3 users not shown)
Line 1: Line 1:
--[[User:Scraig@pitt.edu|Scotty]] 12:53, 19 September 2006 (EDT)
 
 
 
== Learning from Problem Solving while Observing Worked Examples ==
 
== Learning from Problem Solving while Observing Worked Examples ==
 
  ''Scotty Craig, Soniya Gadgil, Kurt VanLehn, and Micki Chi''
 
  ''Scotty Craig, Soniya Gadgil, Kurt VanLehn, and Micki Chi''
=== Summary ===
+
=== Summary Table ===
Start Date: 9-1-06
+
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
 
+
| '''PI''' || Scotty Craig
End Date: 9-1-07
+
|-
 
+
| '''Other Contributers''' || Robert N. Shelby (USNA), Brett van de Sande (Pitt)
LearnLab Site(s): USNA
+
|-
 
+
| '''Study Start Date''' || Sept. 1, 2006
Number of Students: 64
+
|-
 
+
| '''Study End Date''' || Aug. 31, 2007
Particpant Hours:128
+
|-
 
+
| '''LearnLab Site''' || USNA
Data in DataShop? No. Andes data still not compatible with Datashop.
+
|-
 +
| '''LearnLab Course''' || Physics
 +
|-
 +
| '''Number of Students''' || ''N'' = 64
 +
|-
 +
| '''Total Participant Hours''' || 128 hrs.
 +
|-
 +
| '''DataShop''' || Target date: April 30, 2007
 +
|}
 +
<br>
  
 
=== Abstract ===
 
=== Abstract ===
This research project investigated why students learn from [[collaboratively observing]] [[example]]s. Previous laboratory research has shown that learners who watch a video of a problem solving tutoring session while collaboratively solving the same problems with a partner learn significantly more than learners that watched the video and solved the problems alone (Chi, Hausmann, & Roy, under revision). In this study, the [[robustness]] of this effect was tested by seeing if it transfers into the Physics learnlab and the strength of the effect by by testing it against a viewing a worked example.  
+
This research project investigated why students learn from [[collaboratively observing]] [[example]]s in the Phsics LearnLab on the principles of rotational kinematics. The study reported here took this observational learning methodology into the classroom and tested the active observing hypothesis. In doing so, we compared [[collaboratively observing|collaborative observers]] of tutoring videos during problem solving in Andes (Collaboratively observing tutoring condition) against two control conditions that received [[worked examples]]. So the tutoring videos showed an expert human tutor helping undergraduates solve problems, while the [[worked examples]] videos showed the expert tutor solving problems while orally describing the steps and reasoning. The first control condition required pairs of students to collaboratively observe a [[worked examples]] video during problem solving in Andes ([[Collaboratively observing]] [[example]]s condition). The second condition, individually observing examples condition, was comprised of individual students viewing a worked example video alone while problem solving in Andes. Since the [[Andes]] system provides video explanations for the learners on select problems, this control was analogous to the help normally provided in the course. Since both Chi et al. (2008) and Craig et al. (2004) did not find learning gains for individuals observing tutoring, the individually observing of tutoring condition was not taken into the classroom in order to avoid exposing students to an ineffectual learning condition.
  
Students either collaboratively or individually observed videos on the principles of rotational kinematics. The videos were presented either a tutoring session or a worked example. The tutoring videos showed an expert human tutor working with undergraduates taking an introductory physics course. The worked example videos consisted of the expert tutor solving the rotational kinematics problems while orally describing the steps and reasoning.  The Andes system is used throughtout the experiment both as the backdrop for the two sets of videos and by the students who solved [[Andes]] problems both during training and as transfer assesments. However, since this study was conducted in the learnlab, the condition where an individual observed the tutoring session was eliminated because previous lab studies have not shown this contrast to be effective. So, this left three conditions for the current study: collaboratively observing tutoring, Collaboratively observing a worked example, and individually observing a worked example.
+
In the experimental conditions, students collaboratively observed videos. The videos showed either a tutoring session or worked examples.   In the control condition, students viewed the [[worked examples]] video alone, without a collaborating peer. The same problems were shown in all videos.  The [[Andes]] system was used throughtout the experiment both as the backdrop for the two sets of videos and by the students who solved Andes problems both during training and as transfer assesments. In summary, three conditions for the current study were: collaboratively observing tutoring, [[collaboratively observing]] [[worked examples]], and [[individually observing]] [[worked examples]].
  
 +
Analyses have been conducted on immediate learning (Normal pretest/posttest, near transfer) and retention measures. No differences between groups were found for immediate learning. While these immediate learning measures did not display group differences, our long-term and transfer learning measures showed consistent differences in favor of collaboratively observing tutoring.
  
 
=== Glossary ===
 
=== Glossary ===
Line 29: Line 37:
  
 
=== Independent variables ===
 
=== Independent variables ===
The current study varied both number of observers and type of video observed. The multiple-observer variable consisted of two participants observing a video while problem solving or an individual participant watching a video while problem solving. Information presentation format was used to manipulate the example type variable. Participants watched one of two videos. They either watched an expert worked example of Andes problem solving that provided the solution steps for Andes problems along with information on why the steps where needed. Alternatively, they watched a tutoring session where a human tutor worked with a tutee to help solve the Andes problems. Since this study was conducted in the learnlab, the condition where an individual observed the tutoring session was eliminated because previous lab studies have not shown this contrast to be effective.
+
The current study varied both number of observers and type of video observed. The multiple-observer variable consisted of two participants observing a video while problem solving or an individual participant watching a video while problem solving -- see [[collaboration]]. Information presentation format was used to manipulate the example type variable. Participants watched one of two videos. They either watched an expert worked example of Andes problem solving that provided the solution steps for Andes problems along with information on why the steps where needed. Alternatively, they watched a tutoring session where a human tutor worked with a tutee to help solve the Andes problems -- see [[vicarious learning]]. Since this study was conducted in the learnlab, the condition where an individual observed the tutoring session was eliminated because previous lab studies have not shown this contrast to be effective.
 +
 
 +
'''Figure 1. '''Schreenshot of Andes system. <br>
 +
[[Image:CraigetalAndesscreenshot1.JPG]]
  
 
=== Hypothesis ===
 
=== Hypothesis ===
A dialogue hypothesis for collaboratively observing while problem solving from worked examples would be that viewing the expert tutoring session would produce more learning (normal or robust) than viewing a content equivalent condition of expert problem solving. However, an alternative (Content equivalency hypothesis) would be that since the expert tutoring session and the expert worked example both provide good learning conditions with the same content they should both produce mastery of the material (Klahr & Nigam, 2004). Process data collected in this study will help to tease out these hypotheses.
+
There are two contrasting hypotheses being tested in this design. The hypothesis that we have spent the most time with in the current paper is the active observing hypothesis. The active observing hypothesis would predict that the learners in the collaboratively observing tutoring condition would outperform other conditions because  of the highly dynamic tutoring session So, the tutoring videos would contain dialogue features (e.g. turn taking, pauses, and affect) and expert tutoring elements (e.g. corrections and scaffolding) that would promote more active engagement with the video material than the passive information display from the worked example. <br>
 +
Collaboratively observing tutoring > Collaboratively observing examples = Individually observing tutoring      (1)
 +
<br>However, a potential alternative hypothesis (content equivalency hypothesis) is that the content what really matters. Since we have given learners equal content, then the method in which the material is presented should not matter (Klahr & Nigam, 2004). <br>
 +
Collaboratively observing tutoring = Collaboratively observing examples = Individually observing tutoring      (2)
  
 
=== Dependent variables ===
 
=== Dependent variables ===
Line 44: Line 58:
  
 
=== Results ===
 
=== Results ===
Some preliminary analyses were conducted on our transfer multiple choice data. So far, an analysis of the data has yielded significant learning gains between pretest to posttest, ''F'' (1,65) = 14.987, ''p'' <.001 with a proportional ''M'' =.56 and ''M'' = .66 respectively. To date, no significant differences have been found among conditions on the assesment data.
+
===='''Immediate Learning measures'''====
 +
Preliminary analyses have been conducted on immediate learning (MC pretest/posttest, Andes problem solving) and long-term retention measures. An analysis of the data has yielded significant learning gains between pretest to posttest, ''F'' (1, 65) = 14.99, ''p'' <.001 with a proportional ''M'' =.57 and ''M'' = .67 respectively. No differences between groups were found for immediate learning.<br>
 +
[[Image:Craigetaltable1.JPG]]
 +
 
 +
===='''Long term learning measures''' (robust learning)====
 +
Long term retention data. An ANOVA was performed on the participants’ long term retention data to determine differences among groups. This analysis revealed a significant effect of among conditions, F(2, 59) = 3.44, p < .05, Eta sqared = 0.104; pairwise comparisons using LSD tests for main effects revealed that participants in the collaboratively observing tutoring conditions significantly outperformed learners in the collaboratively observing examples condition (p < .05) and the individually observing examples condition (p < .05).
 +
 
 +
===='''Near transfer data.''' (robust learning)====
 +
An ANOVA was conducted on the participants’ near transfer data to determine differences among groups. This analysis revealed a significant effect of among conditions, F(2, 59) = 4.39, p < .05, Eta sqared  = 0.129. Pairwise comparisons using LSD tests for main effects revealed that participants in the collaboratively observing tutoring condition significantly outperformed participants in both the collaboratively observing examples condition (p < .01) and the individually observing examples condition (p < .05).
 +
 
 +
===='''Far transfer data''' (robust learning).====
 +
An ANOVA was run on the participants far transfer data to determine differences among groups. This analysis revealed a significant effect of among conditions, F(2, 59) = 4.89, p < .05, Eta sqared  = 0.142. Pairwise comparisons using LSD tests for main effects revealed that participants in the collaboratively observing tutoring condition significantly outperformed participants in both the collaboratively observing examples condition (p < .01) and the individually observing examples condition (p < .05).
 +
 
 +
[[Image:Craigetaltable2.JPG]]
 +
 
 +
<br>
  
 
=== Explanation ===
 
=== Explanation ===
Line 52: Line 81:
  
 
=== Annotated bibliography ===
 
=== Annotated bibliography ===
 +
*  Craig, S., Vanlehn, K., Gadgil, S., & Chi, M. (2007). Learning from Collaboratively Observing during problem solving with videos. AIED07: 13th International Conference on Artificial Intelligence in Education, Los Angeles, CA. [http://andes3.lrdc.pitt.edu/~scraig/publications/AIED_Observer_learning_LearnLab.pdf]
  
 
=== References ===
 
=== References ===

Latest revision as of 16:35, 7 October 2008

Learning from Problem Solving while Observing Worked Examples

Scotty Craig, Soniya Gadgil, Kurt VanLehn, and Micki Chi

Summary Table

PI Scotty Craig
Other Contributers Robert N. Shelby (USNA), Brett van de Sande (Pitt)
Study Start Date Sept. 1, 2006
Study End Date Aug. 31, 2007
LearnLab Site USNA
LearnLab Course Physics
Number of Students N = 64
Total Participant Hours 128 hrs.
DataShop Target date: April 30, 2007


Abstract

This research project investigated why students learn from collaboratively observing examples in the Phsics LearnLab on the principles of rotational kinematics. The study reported here took this observational learning methodology into the classroom and tested the active observing hypothesis. In doing so, we compared collaborative observers of tutoring videos during problem solving in Andes (Collaboratively observing tutoring condition) against two control conditions that received worked examples. So the tutoring videos showed an expert human tutor helping undergraduates solve problems, while the worked examples videos showed the expert tutor solving problems while orally describing the steps and reasoning. The first control condition required pairs of students to collaboratively observe a worked examples video during problem solving in Andes (Collaboratively observing examples condition). The second condition, individually observing examples condition, was comprised of individual students viewing a worked example video alone while problem solving in Andes. Since the Andes system provides video explanations for the learners on select problems, this control was analogous to the help normally provided in the course. Since both Chi et al. (2008) and Craig et al. (2004) did not find learning gains for individuals observing tutoring, the individually observing of tutoring condition was not taken into the classroom in order to avoid exposing students to an ineffectual learning condition.

In the experimental conditions, students collaboratively observed videos. The videos showed either a tutoring session or worked examples. In the control condition, students viewed the worked examples video alone, without a collaborating peer. The same problems were shown in all videos. The Andes system was used throughtout the experiment both as the backdrop for the two sets of videos and by the students who solved Andes problems both during training and as transfer assesments. In summary, three conditions for the current study were: collaboratively observing tutoring, collaboratively observing worked examples, and individually observing worked examples.

Analyses have been conducted on immediate learning (Normal pretest/posttest, near transfer) and retention measures. No differences between groups were found for immediate learning. While these immediate learning measures did not display group differences, our long-term and transfer learning measures showed consistent differences in favor of collaboratively observing tutoring.

Glossary

See Craig Observing tutoring Glossary

Research question

How is robust learning affected by collaboratively versus individually observing different types of worked examples?

Independent variables

The current study varied both number of observers and type of video observed. The multiple-observer variable consisted of two participants observing a video while problem solving or an individual participant watching a video while problem solving -- see collaboration. Information presentation format was used to manipulate the example type variable. Participants watched one of two videos. They either watched an expert worked example of Andes problem solving that provided the solution steps for Andes problems along with information on why the steps where needed. Alternatively, they watched a tutoring session where a human tutor worked with a tutee to help solve the Andes problems -- see vicarious learning. Since this study was conducted in the learnlab, the condition where an individual observed the tutoring session was eliminated because previous lab studies have not shown this contrast to be effective.

Figure 1. Schreenshot of Andes system.
CraigetalAndesscreenshot1.JPG

Hypothesis

There are two contrasting hypotheses being tested in this design. The hypothesis that we have spent the most time with in the current paper is the active observing hypothesis. The active observing hypothesis would predict that the learners in the collaboratively observing tutoring condition would outperform other conditions because of the highly dynamic tutoring session So, the tutoring videos would contain dialogue features (e.g. turn taking, pauses, and affect) and expert tutoring elements (e.g. corrections and scaffolding) that would promote more active engagement with the video material than the passive information display from the worked example.
Collaboratively observing tutoring > Collaboratively observing examples = Individually observing tutoring (1)
However, a potential alternative hypothesis (content equivalency hypothesis) is that the content what really matters. Since we have given learners equal content, then the method in which the material is presented should not matter (Klahr & Nigam, 2004).
Collaboratively observing tutoring = Collaboratively observing examples = Individually observing tutoring (2)

Dependent variables

  • Transfer, immediate: After exposure to the treatment, students completed three transfer problems in Andes. These problems will test the same concepts from training in new situations that require implementation of the problems in new ways.
  • Normal post-test: Students were given a 12 item multiple choice pretest and posttest that taps into their ability to apply the principles of rotational kinematics to new situations. This served as a measure of immediate learning for the study.
  • Homework as long-term retention and transfer items: After training, students completed their regular homework problems using Andes. Students could do them whenever they want, but most normally complete them just before the exam. The homework problems were divided based on similarity to the training problems. Homework for both similar (near transfer) and dissimilar (far transfer) problems will be analyzed.
  • Accelerated future learning: The training was on Rotational kinematics, and it was followed in the course by a unit on Rotational Dynamics. Andes log files from this homework will be analyzed as a measure of acceleration of future learning.

Results

Immediate Learning measures

Preliminary analyses have been conducted on immediate learning (MC pretest/posttest, Andes problem solving) and long-term retention measures. An analysis of the data has yielded significant learning gains between pretest to posttest, F (1, 65) = 14.99, p <.001 with a proportional M =.57 and M = .67 respectively. No differences between groups were found for immediate learning.
Craigetaltable1.JPG

Long term learning measures (robust learning)

Long term retention data. An ANOVA was performed on the participants’ long term retention data to determine differences among groups. This analysis revealed a significant effect of among conditions, F(2, 59) = 3.44, p < .05, Eta sqared = 0.104; pairwise comparisons using LSD tests for main effects revealed that participants in the collaboratively observing tutoring conditions significantly outperformed learners in the collaboratively observing examples condition (p < .05) and the individually observing examples condition (p < .05).

Near transfer data. (robust learning)

An ANOVA was conducted on the participants’ near transfer data to determine differences among groups. This analysis revealed a significant effect of among conditions, F(2, 59) = 4.39, p < .05, Eta sqared = 0.129. Pairwise comparisons using LSD tests for main effects revealed that participants in the collaboratively observing tutoring condition significantly outperformed participants in both the collaboratively observing examples condition (p < .01) and the individually observing examples condition (p < .05).

Far transfer data (robust learning).

An ANOVA was run on the participants far transfer data to determine differences among groups. This analysis revealed a significant effect of among conditions, F(2, 59) = 4.89, p < .05, Eta sqared = 0.142. Pairwise comparisons using LSD tests for main effects revealed that participants in the collaboratively observing tutoring condition significantly outperformed participants in both the collaboratively observing examples condition (p < .01) and the individually observing examples condition (p < .05).

Craigetaltable2.JPG


Explanation

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s hypothesis is that robust learning occurs when two conditions are met. Specific explanations for the current study follow.

  • The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work. In this study, the collaboration conditions could comprise the learning-by-doing paths where learners can work together to complete the Andes problems or the paired learners could rely on the video as their information providing agent and simply copy the steps. Alternatively the participants in the solo condition would have to rely exclusively on the video for information and thus rely on more direct copying of steps thus allowing another agent (the video) to do most of the work. In this case, both learning conditions offer the alternate copying path. However, copying could differ in frequency and be more likely to be discouraged in the collaborative condition due to the more social nature of the task.
  • The student takes the learning-by-doing path unless it becomes too difficult. This study attempts to control the student’s path choice by presenting them with the tutorial dialogue that could encourage communication or an expert worked example that gives a walk through of the problem without the dialogue interaction. So, in the conditions where students are more likely to take the learning-by-doing path (the tutoring dialogue conditions), they are more likely to learn more, as compared to the conditions where they are more likely to take an alternative path (in the expert worked example conditions).

Annotated bibliography

  • Craig, S., Vanlehn, K., Gadgil, S., & Chi, M. (2007). Learning from Collaboratively Observing during problem solving with videos. AIED07: 13th International Conference on Artificial Intelligence in Education, Los Angeles, CA. [1]

References

  • Chi, M. T. H., Hausmann, R. G. M., & Roy, M. (in press). Learning from observing tutoring collaboratively: Insights about tutoring effectiveness from vicarious learning. Cognitive Science.
  • Craig, S. D., Driscoll, D., & Gholson, B. (2004). Constructing knowledge from dialog in an intelligent tutoring system: Interactive learning, vicarious learning, and pedagogical agents. Journal of Educational Multimedia and Hypermedia, 13, 163-183. [2]
  • Gholson, B. & Craig, S. D. (2006). Promoting constructive activities that support vicarious learning during computer-based instruction. Educational Psychology Review, 18, 119-139. [3]
  • Klahr, D. & Nigam, M. (2004). The equivalence of learning paths in early science instruction: Effects of direct instruction and discovery learning. Psychological Science, 15, 661-667.

Connections

This project shares features with the following research projects:

Collaboration during learning

Worked examples and learning