Difference between revisions of "Does learning from worked-out examples improve tutored problem solving?"

From LearnLab
Jump to: navigation, search
(→‎Independent variables)
m (Reverted edits by Kamearobinson (Talk); changed back to last version by Ron-Salden)
 
(133 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 
== Does learning from worked-out examples improve tutored problem solving? ==
 
== Does learning from worked-out examples improve tutored problem solving? ==
−
  ''Alexander Renkl, Vincent Aleven, & Ron Salden''
+
  ''Alexander Renkl, Rolf Schwonke, Vincent Aleven, & Ron Salden''
  
 
=== Abstract ===
 
=== Abstract ===
−
Although problem solving supported by Cognitive Tutors has been shown to be successful in fostering initial acquisition of cognitive skill, this approach does not seem to be optimal with respect to focusing the learner on the domain principles to be learned. In order to foster a deep understanding of domain principles and how they are applied in problem solving, we combine the theoretical rationales of Cognitive Tutors and example-based learning. Especially, we address the following main hypotheses: (1) Enriching a Cognitive Tutor unit with examples whose worked-out steps are gradually faded leads to better learning; (2) individualizing the fading procedure based on the quality of self-explanations that the learners provide further improves learning; (3) using free-form self-explanations is more useful in this context as compared to the usual menu-based formats; (4) learning can be enhanced further by providing previously self-explained examples – including the learner’s own self-explanations – as support at problem-solving impasses. We address these research questions by preparatory lab experiments and subsequent field experiments in the Geometry LearnLab.  
+
In our research we combine two educational research branches. The first concerns the Cognitive Tutor research branch which deals with offering students the possibility to attain and extend their skills through means of a computerized curriculum of problems. The second branch concerns the research done on worked-out examples and the fading of worked-out steps. Because students’ prior knowledge on novice elements is usually low it is widely believed presenting a worked-out example might give the student more grasp on the novice elements. By presenting more information about the novice elements the student will not have to bridge the gap between his prior knowledge and the novice elements entirely by himself.
  
−
We have already performed two laboratory experiment on research question 1. Detailled analyses of the process data are still in progress. Up to now, we found the following results with respect to learning outcomes and time-on-task (i.e., learning time). In a first experiment, we compared a Cognitive Tutor unit with worked-out examples and and one without examples; both versions comprised self-explanation prompts. We found no differences in the learning outcome variables of conceptual understanding and procedural skills (transfer). However, the example-enriched tutor led to significantly shorted learning times. We also found a significant advantage with respect to an efficiency measure relating the learning time to learning outcomes. Informal observations showed that the participants (German students) were in part confused that the solution was already given in the example condition ("What should we exactly do?"). As a consequence, we informed the students more fully about the respective Cognitive Tutor environments to be studied in a second experiment. In addition, we collected thinking aloud data (yet to be analyzed). We found significant advantages of the example condition with respect to conceptual knowledge, learning time (less time), and efficiency of learning. With respect to procedural skills no differences were observed.
+
However, as the students progresses through a curriculum his knowledge will grow and fully worked-out examples might give him redundant information. Hence, as the curriculum advances the number of worked-out steps in a problem is gradually decreased until all steps are left blank (i.e., problem solving). By continually fading the worked-out steps the student will have more optimal learning instances than either from only fully worked-out problems or only problem solving in the curriculum. Our studies address the main hypothesis which states that an example enriched Cognitive Tutor can create more deep conceptual understanding.
 +
 
 +
In our recent In Vivo Study ('''Study 4''') we compared a control condition with two experimental conditions which faded worked-out examples. In the ''control condition'' the students had to solve problems and enter explanations for each step as they are used to do. In the ''fixed fading condition'' the same preset fading of worked-out steps was applied to all students. In the ''adaptive fading condition'' the fading of the worked-out steps was based on the individual student's progress as measures on their problem solving and their reasoning. The data of this study have been collected and are currently being processed to the Datashop after which data analysis will commence.
  
 
=== Background and Significance ===
 
=== Background and Significance ===
−
The background of this research is twofold. (1) The very successful approach of Cognitive Tutors (Anderson, Corbett, Koedinger, & Pelletier, 1995; Koedinger, Anderson, Hadley, & Mark, 1997) is taken up. These computer-based tutors provide individualized support for learning by doing (i.e., solving problems) by selecting appropriate problems to-be-solved, by providing feedback and problem-solving hints, and by on-line assessment of the student’s learning progress. Cognitive Tutors individualize the instruction by selecting problems based on a model of the students’ present knowledge state that is constantly updated, through a Bayesian process called “knowledge tracing” (Corbett & Anderson, 1995). A restriction of learning in Cognitive Tutor is that conceptual understanding is not a major learning goal. (2) The research tradition on worked-out examples rooted in Cognitive Load Theory (Sweller, van Merrienboer, & Paas, 1998) and, more specifically, the instructional model of example-based learning by Renkl and Atkinson (in press) are taken up in order to foster skill acquisition that is found in deep conceptual udnerstanding. By presenting examples instead of problems to be solved in the beginning of a learning sequence, the learner have more attentential capacity availabel in order to self-explain and thus deepen their understanding of problem solutions.
+
The background of this research is twofold. (1) The very successful approach of Cognitive Tutors (Anderson, Corbett, Koedinger, & Pelletier, 1995; Koedinger, Anderson, Hadley, & Mark, 1997) is taken up. These computer-based tutors provide individualized support for learning by doing (i.e., solving problems) by selecting appropriate problems to-be-solved, by providing feedback and problem-solving hints, and by on-line assessment of the student’s learning progress. Cognitive Tutors individualize the instruction by selecting problems based on a model of the students’ present knowledge state that is constantly updated, through a Bayesian process called “knowledge tracing” (Corbett & Anderson, 1995). Although problem solving supported by [[cognitive tutor]]s has been shown to be successful in fostering initial acquisition of cognitive skill, this approach does not seem to be optimal with respect to focusing the learner on the domain principles to be learned.  
 +
(2) The research tradition on worked-out examples rooted in Cognitive Load Theory (Sweller, van Merriënboer, & Paas, 1998) and, more specifically, the instructional model of example-based learning by Renkl and Atkinson (in press) are taken up in order to foster skill acquisition that is found in deep conceptual understanding. By presenting examples instead of problems to be solved in the beginning of a learning sequence, the learner have more attentional capacity available in order to self-explain and thus deepen their understanding of problem solutions.
 +
 
 +
In order to foster a deep understanding of domain principles and how they are applied in problem solving, we combine the theoretical rationales of Cognitive Tutors and example-based learning.
 +
 
 +
Especially, we address the following main hypotheses:
 +
# Enriching a Cognitive Tutor unit with [[example]]s whose worked-out steps are gradually faded leads to better learning.
 +
# Individualizing the fading procedure based on the quality of self-explanations that the learners provide further improves learning.
 +
# Using free-form self-explanations is more useful in this context as compared to the usual menu-based formats.
 +
# Learning can be enhanced further by providing previously self-explained examples – including the learner’s own self-explanations – as support at problem-solving [[impasse]]s.
  
−
This project is in several respects of signficance:
+
This project is in several respects of significance:
  
 
(1) Presently, the positive effects of examples were shown in comparison to unsupported problem solving. We aim to show that example study is also superior to supported problem solving in the very beginning of a learning sequence.
 
(1) Presently, the positive effects of examples were shown in comparison to unsupported problem solving. We aim to show that example study is also superior to supported problem solving in the very beginning of a learning sequence.
Line 16: Line 27:
 
(2) The Cognitive Tutor approach can be enhanced by ideas from research on example-based learning.
 
(2) The Cognitive Tutor approach can be enhanced by ideas from research on example-based learning.
  
−
(3) The example-based learning approach can be enriched by individualizing instructinal procedures such as fading.
+
(3) The example-based learning approach can be enriched by individualizing instructional procedures such as fading.
  
 
=== Glossary ===
 
=== Glossary ===
−
To be developed, but will probably include:
 
  
−
''Learning by worked-out examples''
+
[[:Category:Salden Learning-from-Examples|Salden Learning-from-Examples Glossary]]
  
−
''Learning by problem solving''
+
[[Learning by worked-out examples]]
  
−
''Self-explanation''
+
[[Learning by problem solving]]
  
−
''Fading''
+
[[Self-explanation]]
 +
 
 +
[[Fading]]
  
 
=== Research question ===
 
=== Research question ===
Line 41: Line 53:
 
(b) Cognitive Tutors with intially worked-out examples, then partially worked-out examples, and finally problem to be solved.
 
(b) Cognitive Tutors with intially worked-out examples, then partially worked-out examples, and finally problem to be solved.
  
−
Although self-explanation prompts are a typical "ingredient" of example-based learning, but not of learning by problem solving, such prompts were included in both conditions. Thereby, the potential effects can be clearly attributed to the presence or absense of example study.
+
(b1) The fading occurs according to a fixed procedure which is used for all students.
 +
 
 +
(b2) The fading occurs adaptively based on the knowledge ''and'' reason tracing of the individual student's progress.
 +
 
 +
(c) Previously performed problems are accessible as hints. Meaning that the regular textual hints are replaced by screenshots of the previously performed problems.
 +
 
 +
Although self-explanation prompts are a typical "ingredient" of example-based learning, but not of learning by problem solving, such prompts were included in all conditions of our studies. Thereby, the potential effects can be clearly attributed to the presence or absense of example study.
 +
The exception to this is Study 6 which includes two Untutored conditions. For more information on this study please scroll down to the '''Further Information''' section.
 +
 
 +
=== Dependent variables ===
 +
1) [[Normal post-test]] of [[procedural|procedural knowledge]]: measured by the students' performance on new problems dealing with the content they learned in the Cognitive Tutor.
 +
 
 +
1) [[Transfer]] test assessing [[conceptual knowledge]]: measured by a variety of questions on the post test that are different in format, non-isomorphic, from those used in training. These include drawing problems, multiple questions and open questions concerning the specific geometric principles the students were exposed to in the Cognitive Tutor.
 +
 
 +
3) [[Accelerated future learning|Acceleration of future learning]] (in future experiments): we plan to use a related unit to the angles and/or circles units which follows each of these units in the curriculum.
 +
 
 +
4) Learning time: measured in total time to complete the unit in the Cognitive Tutor.
 +
 
 +
5) Efficiency of learning (relating learning time to learning outcomes): based on the mental efficiency measure developed by Paas and van Merriënboer (1993).
 +
 
 +
=== Hypotheses & Results ===
 +
The provision of example in Cognitive Tutors should lead to better conceptual understanding and, thereby, transfer performance. In addition, examples in Cognitive Tutors should reduce learning time.
 +
 
 +
On the whole, the present results confirm the hypotheses with respect to conceptual knowledge and learning time. The expected effects on transfer were not found.
 +
 
 +
===Study 1 (lab study at Freiburg, Geometry Cognitive Tutor) ===
 +
*''Summary''
 +
**Lab Study: 8th grade and 9th grade students from a German high school in Freiburg
 +
**Domain: translated Circles Unit in the Geometry Cognitive Tutor
 +
**Start Date: March 1, 2006
 +
**End Date: March 31, 2006
 +
**Number of Students: 50
 +
**Participant Hours: 1.5
 +
**Data in Datashop: Yes
 +
 
 +
*The students were randomly assigned to one of two conditions:
 +
**Problem Solving Condition: In this control condition students solved answer steps and entered explanations on all problems
 +
[[Image:Freiburg_Summer_2006_-_problem_solving_versus_fixed_fading_examples_study_-_Problem_condition-small.jpg]]
 +
**Worked Example Condition: In this experimental condition students were first presented with problems that had worked out (i.e., filled in) answer steps but still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.
 +
[[Image:Freiburg_Summer_2006_-_problem_solving_versus_fixed_fading_examples_study_-_Example_condition-small.jpg]]
 +
 
 +
*''Findings''
 +
**No overall effect of experimental condition on students' conceptual and procedural knowledge on the post-test: ''t'' < 1.
 +
**However, about the same learning outcomes were achieved in shorter learning times in the example-enriched Cognitive Tutor: ''t''(48) = -3.11, p < .001 (one-tailed), ''r'' = .41.
 +
**Accordingly, the efficiency of learning was superior in this latter learning condition: ''t''(48) = 1.73, ''p'' < .05 (one-tailed), ''r'' = .24 for conceptual knowledge, and ''t''(48) = 1.82, ''p'' < .05 (one-tailed), ''r'' = .25 for the acquisition of transferable knowledge.
 +
 
 +
===Study 2 (lab study at Freiburg, Geometry Cognitive Tutor) ===
 +
*''Summary''
 +
**Lab Study: 9th grade and 10th grade students from a German high school in Freiburg
 +
**Domain: translated Circles Unit in the Geometry Cognitive Tutor
 +
**Start Date: April 1, 2006
 +
**End Date: April 31, 2006
 +
**Number of Students: 30
 +
**Participant Hours: 1.5
 +
**Data in Datashop: Yes
 +
 
 +
*The students were randomly assigned to one of two conditions:
 +
**Problem Solving Condition: In this control condition, students solved answer steps and entered explanations on all problems. See Study 1 for a screenshot of this condition.
 +
**Worked Example Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations. See Study 1 for a screenshot of this condition.
 +
 
 +
*''Findings''
 +
**With regard to conceptual understanding part of the transfer post-test, an advantage of the example condition over the problem condition was found: ''t''(28) = 1.85, ''p'' < .05 (one-tailed), ''r'' = 0.33.
 +
**There were no significant differences on other, procedural, transfer items: ''t'' < 1.
 +
**Similar to Study 1, students in the example condition spent significantly less time than students in the problem solving condition: ''t''(28) = -3.14, ''p'' < .001 (one-tailed), ''r'' = 0.51.
 +
**Hence, on a measure of robust learning efficient (i.e., relating performance in terms of the acquisition of conceptual knowledge to the effort in terms of time on task) a large effect was obtained: ''r'' = 0.55, ''t''(28) = 3.48, ''p'' < .001.
 +
 
 +
===Study 3 (lab study at CMU, Geometry Cognitive Tutor) ===
 +
*''Summary''
 +
**Lab Study: 8th grade and 9th grade students in rural Pennsylvannia schools
 +
**Domain: Circles Unit in the Geometry Cognitive Tutor
 +
**Start Date: October 1, 2006
 +
**End Date: November 30, 2006
 +
**Number of Students: 45
 +
**Participant Hours: 2
 +
**Data in Datashop: Yes
 +
 
 +
*The students were randomly assigned to one of two conditions:
 +
**Problem Solving Condition: In this control condition, students solved answer steps and entered explanations on all problems. See Study 1 for a screenshot of this condition.
 +
**Worked Example Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations. See Study 1 for a screenshot of this condition.
 +
 
 +
*''Findings''
 +
**No overall effect of experimental condition on students' conceptual and procedural knowledge on the post-test: ''t'' < 1.
 +
**No difference in time to complete the Circles Unit in the Tutor: ''t'' < 1.
 +
 
 +
===Study 4 (In Vivo study at CMU, Geometry Cognitive Tutor) ===
 +
*''Summary''
 +
**In Vivo Study: 10th grade geometry classes in rural Pennsylvannia high school
 +
**Domain: Angles Unit in the Geometry Cognitive Tutor
 +
**Start Date: January 8, 2007
 +
**End Date: March 9, 2007
 +
**Number of Students: 51
 +
**Participant Hours: 5
 +
**Data in Datashop: Yes
 +
 
 +
*The students were randomly assigned to one of three conditions:
 +
**Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems.
 +
[[Image:January_2007_-_adaptive_fading_study_CWCTC_-_Problem_solving_condition-small.jpg]]
 +
**Fixed Fading of Worked Examples Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.
 +
[[Image:January_2007_-_adaptive_fading_study_CWCTC_-_Fixed_fading_condition-small.jpg]]
 +
**Adaptive Fading of Worked Examples Condition: This experimental condition is similar to the Fixed Fading condition but differs in the fact that the fading of the filled in answer steps is based on the individual student's performance on both the answer and the reason steps.
 +
See screenshot of the Fixed Fading condition, the interface was identical in both condition.
 +
 
 +
*''Findings''
 +
**Positive effect of the Adaptive Fading Condition over the Problem Solving Condition on the delayed post-test: ''t''(20) = 2.15, ''p'' < .05, ''d'' = .96.
 +
**No difference in time to complete the Circles Unit in the Tutor: ''t'' < 1.
 +
 
 +
===Study 5 (lab study at Freiburg, Geometry Cognitive Tutor) ===
 +
*''Summary''
 +
**Lab Study: 9th grade and 10th grade students from a German high school in Freiburg
 +
**Domain: translated Circles Unit in the Geometry Cognitive Tutor
 +
**Start Date: March 19, 2007
 +
**End Date: March 26, 2007
 +
**Number of Students: 57
 +
**Participant Hours: 2
 +
**Data in Datashop: Yes
 +
 
 +
*The students were randomly assigned to one of three conditions:
 +
**Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems.
 +
**Fixed Fading of Worked Examples Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.
 +
**Adaptive Fading of Worked Examples Condition: This experimental condition is similar to the Fixed Fading condition but differs in the fact that the fading of the filled in answer steps is based on the individual student's performance on both the answer and the reason steps.
  
−
=== Hypothesis ===
+
*''Findings''
−
For these well-prepared students, self-explanation should not too difficult. That is, the instruction should be below the students’ zone of proximal development. Thus, the learning-by-doing path (self-explanation) should elicit more robust learning than the alternative path (instructional explanation) wherein the student does less work.  
+
**A planned contrast comparing the adaptive fading condition with the problem solving + fixed fading conditions revealed higher transfer performance for the adaptive fading condition on the regular post-test: ''F''(1, 54) = 5.05, ''p'' < .05, ''η²'' = .09. This effect was replicated on the delayed post-test: ''F''(1, 54) = 4.42, ''p'' < .05, ''η²'' = .08.
 +
**There were no differences in time spent on either of the post-tests: ''F''s < 1.
  
−
As a manipulation check on the utility of the explanations in the complete examples, we hypothesize that instructional explanation condition should produce more robust learning than the no-explanation condition.
+
===Study 6 (lab study at CMU, Geometry Cognitive Tutor) ===
 +
*''Summary''
 +
**Lab Study: 8th, 9th and 10th grade students in rural Pennsylvannia school
 +
**Domain: Circles Unit in the Geometry Cognitive Tutor
 +
**Start Date: April 18, 2007
 +
**End Date: April 19, 2007
 +
**Number of Students: 62
 +
**Participant Hours: 2
 +
**Data in Datashop: Yes
  
−
=== Dependent variables & Results ===
+
*The students were randomly assigned to one of 4 conditions:
−
* ''Near transfer, immediate'': During training, examples alternated with problems, and the problems were solved using Andes.   Each problem was similar to the example that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). The log data were analyzed and assistance scores (sum of errors and help requests) were calculated.  There was a main effect of Study Strategy on assistance score, reflecting higher scores for the paraphrase condition than the self-explanation condition.
+
**Tutored Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems in the standard Cognitive Tutor.
 +
**Tutored Worked Examples Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.
 +
**Untutored Problem Solving Condition: In this experimental condition students solved answer steps and entered explanations on all problems in a stripped down version of the Cognitive Tutor. Students could NOT ask for hints and only after entering all answer and reason steps could they receive corrective feedback. They could still browse the Glossary while working on a problem.
 +
[[Image:Screenshot-notutoring_problem_solving.jpg]]
 +
**Untutored Worked ExamplesProblem Solving Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.
 +
Furthermore, students were working in a stripped down version of the Cognitive Tutor. Students could NOT ask for hints and only after entering all answer and reason steps could they receive corrective feedback. They could still browse the Glossary while working on a problem.
 +
[[Image:Screenshot-notutoring_worked_examples.jpg]]
  
−
* ''Near transfer, retention'': On the student’s regular mid-term exam, one problem was similar to the training. Since this exam occurred a week after the training, and the training took place in just under 2 hours, the student’s performance on this problem is considered a test of retention. Results on the measure were mixed. While there were no reliable main effects or interactions, the the complete self-explanation group was marginally higher than the complete paraphrase condition (LSD, p = .064).
+
*''Findings''
 +
**No overall effect of experimental condition on students' conceptual and procedural knowledge on the post-test: ''t'' < 1.
 +
**No difference in time to complete the Circles Unit in the Tutor: ''t'' < 1.
  
−
* ''Near and far transfer'': After training, students did their regular homework problems using Andes.  Students did them whenever they wanted, but most completed them just before the exam.  The homework problems were divided based on similarity to the training problems, and assistance scores were calculated.  On both similar (near transfer) and dissimilar (far transfer) problems, the results are consistent with self-explanation being more effective than instructional explanation.
+
===Study 7 (In Vivo study at CMU, Geometry Cognitive Tutor) ===
 +
*''Summary''
 +
**In Vivo Study: 10th grade geometry classes in two rural Pennsylvannia high schools
 +
**Domain: Circles Unit in the Geometry Cognitive Tutor
 +
**Start Date: April 30, 2007
 +
**End Date: June 1, 2007
 +
**Number of Students: 104
 +
**Participant Hours: 5
 +
**Data in Datashop: Yes
  
−
* ''Acceleration of future learning'': The training was on magnetic fields, and it was followed in the course by a unit on electrical fields.  Log data from the electrical field homework was analyzed as a measure of acceleration of future learning. Both assistance scores and learning curves of the key principles support the hypothesis that self-explanation is more effective than instructional explanation.
+
*The students were randomly assigned to one of 4 conditions:
 +
**Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems in the standard Cognitive Tutor and received step-by-step textual hints. For a screenshot see the Problem Solving Condition of Study 1.
 +
**Worked Examples in Hints Condition: In this experimental condition, students solved answer steps and entered explanations on all problems in the standard Cognitive Tutor and for each Geometry concept received a worked example as a hint. In contrast to the control condition which had several hints levels for each step, this experimental condition had only one hint level: the worked example.
 +
[[Image:Spring_2007_-_examples_in_hints_study_CWCTC_and_Wilkinsburg_-_Examples_condition-small.jpg]]
  
−
=== Explanation ===
+
*''Findings''
−
This study is part of the [[Interactive_Communication|Interactive Communication cluster]], and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s hypothesis is that robust learning occurs when two conditions are met:
+
**No overall effect of experimental condition on students' conceptual and procedural knowledge on the post-test: ''t'' < 1.
 +
**No difference in time to complete the Circles Unit in the Tutor: ''t'' < 1.
  
−
* The learning event space should have paths that are mostly learning-by-doing along with alternative paths were a second agent does most of the work.  In this study, self-explanation comprises the learning-by-doing path and instructional explanations are ones where another agent (the author of the text) has done most of the work.
 
  
−
* The student takes the learning-by-doing path unless it becomes too difficult. This study tried (successfully, it appears) to control the student’s path choice.  It showed that when students take the learning-by-doing path, they learned more than when they take the alternative path.
+
===Study 8 (In Vivo study at CMU, Geometry Cognitive Tutor) ===
 +
*''Summary''
 +
**In Vivo Study: 10th grade geometry classes in three rural Pennsylvannia high schools
 +
**Domain: Circles Unit in the Geometry Cognitive Tutor
 +
**Start Date: April 15, 2009
 +
**End Date: June 10, 2009
 +
**Number of Students: 151
 +
**Participant Hours: 9-10
 +
**Data in Datashop: No
 +
 
 +
*The students were randomly assigned to one of 3 conditions:
 +
**Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems in the standard Cognitive Tutor.
 +
**All Examples Condition: In this experimental condition, students receive worked-out answer steps for all the required problems. There is no fading of these steps thus the first time students get to solve answer steps is on the remedial problem set.
 +
**Adaptive Fading of Worked Examples Condition: This experimental condition is similar to the All Examples condition but differs in the fact that fading of the filled in answer steps occurs and the fading is based on the individual student's performance on both the answer and the reason steps.
 +
 
 +
 
 +
*''Findings''
 +
**The study is still in progress.
 +
 
 +
 
 +
=== Summary of Findings and Explanation ===
 +
This project belongs to the interactive communication cluster because it investigates a variation of the amount of contribution from the system and from the learner, respectively: Who provides the solution of the initial solution steps?
 +
 
 +
More specifically, this study is about changes in path choices that occur when a tutoring system includes partially worked examples.  The basic idea is that when a tutor relieves a student of most of the work in generating a line by providing part of it, then students are more likely to engage in deep learning to fill in the rest.  However, the instruction must be engineered so that students still become autonomous problem solvers—they eventually can do all the work themselves.
 +
 
 +
In the first German laboratory study, the standard Cognitive Tutor was compared with an example-enriched Cognitive Tutor. While no effects on procedural and conceptual knowledge transfer items were found, the students working with the example-enriched Tutor completed the curriculum faster than the students in the standard Cognitive Tutor. Using the learning time to measure the condition efficiency showed that the example-enriched Tutor obtained higher learning efficiency on the transfer test. Since the German students were inexperienced with the Cognitive Tutor, more detailed instructions were provided in the follow up study. Consequently, the students working on the example-enriched Tutor showed a higher gain on the conceptual knowledge items of the transfer test than the students working with the standard Tutor. Furthermore, similar to the first study the example-enriched Tutor led to significantly shorter learning time than the standard Cognitive Tutor. Lastly, using the learning time to measure efficiency revealed higher learning efficiency for the example-enriched Tutor on the conceptual knowledge items of the transfer test.
 +
In terms of the robust learning framework, these results shows that worked-out examples lead to same level of foundational skills in less time. Furthermore, the second study shows that fading worked-out examples can improve sense-making which consequently leads to better robust learning.
 +
 
  
−
The IC cluster’s hypothesis actually predicts an attribute-treatment interaction (ATI) here.  If some students were under-prepared and thus would find the self-explanation path too difficult, then those students would learn more on the instructional-explanation path.  ATI analyzes have not yet been completed.
 
  
 
=== Annotated bibliography ===
 
=== Annotated bibliography ===
−
* Presentation to the NSF Site Visitors, June, 2006
+
Salden R. J. C. M., Aleven, V., Renkl, A., & Wittwer, J. (2006). Does Learning from Examples Improve Tutored Problem Solving? In 2006 Proceedings of the 28th Annual Meeting of the Cognitive Science Society (pp. 2602), Vancouver, Canada. [http://www.learnlab.org/uploads/mypslc/publications/salden.pdf Link to paper]
−
* Preliminary results were presented to the Intelligent Tutoring in Serious Games workshop, Aug. 2006
+
 
−
* Presentation to the NSF Follow-up Site Visitors, September, 2006
+
Presentation to the PSLC Advisory Board, Fall 2006.
 +
 
 +
Schwonke, R., Wittwer, J., Aleven, V., Salden, R. J. C. M., Krieg, C., & Renkl, A. (2007). Can tutored problem solving benefit from faded worked-out examples? Paper presented at The European Cognitive Science Conference 2007, May 23-27. Delphi, Greece. [http://www.learnlab.org/uploads/mypslc/publications/eurocogsci%202007%20fading%20of%20woe%20in%20cogtutor_rs_jw_ar_rs_jw_rs.doc Link to paper]
 +
 
 +
Salden, R., Aleven, V., & Renkl, A. (2007). Can tutored problem solving be improved by learning from examples? Proceedings of the 29th Annual Conference of the Cognitive Science Society (p. 1847), Nashville, USA.
 +
 
 +
Salden, R., Aleven, V., & Renkl, A., & Schwonke, R. (2008). Worked Examples and the Assistance Dilemma. Paper presented at The American Educational Research Association 2008, March 23-27. New York, USA. [http://www.learnlab.org/research/wiki/images/AERA_2008_paper_-_Salden%2C_Aleven%2C_Renkl_%26_Schwonke.pdf Link to paper]
 +
 
 +
Salden, R., Aleven, V., Schwonke, R., & Renkl, A. (2008, June). Worked Are Worked Examples and Tutored Problem Solving Synergistic Forms of Support? Poster presented at the 8th International Conference of the Learning Sciences (ICLS).
 +
[http://www.learnlab.org/research/wiki/images/6/60/ICLS_2008_poster_Salden%2C_Aleven%2C_Schwonke_%26_Renkl.pdf Link to paper]
 +
 
 +
Salden, R., Aleven, V., & Renkl, A., & Schwonke, R. (2008, July). Worked Examples and Tutored Problem Solving: Redundant or Synergistic Forms of Support? Paper presented at the 30th Annual Meeting of the Cognitive Science Society, July 23-26. Washington DC, USA.
 +
[http://www.learnlab.org/research/wiki/images/e/e8/Fp495-salden.pdf ''Winner of the "Cognition and Student Learning" award'']
 +
 
 +
Schwonke, R., Renkel, A., Krieg, C, Wittwer, J., Aleven, V., Salden, R. J. C. M. (2009). The Worked-example Effect: Not an Artefact of Lousy Control Conditions. ''Computers in Human Behavior, 25'', 258-266.
 +
 
 +
Salden, R. J. C. M., Aleven, V. A. W. M. M., Renkl, A., & Schwonke, R. (2009). Worked examples and tutored problem solving: Redundant or synergistic forms of support? ''Topics in Cognitive Science, 1'', 203-213.
 +
 
 +
Salden, R. J. C. M., Aleven, V. A. W. M. M., Schwonke, R., & Renkl, A. (2009). ''The expertise reversal effect and worked examples in tutored problem solving''. Manuscript submitted for publication.
 +
 
 +
 
  
 
=== References ===
 
=== References ===
−
Anzai, Y., & Simon, H. A. (1979). The theory of learning by doing. ''Psychological Review, 86''(2), 124-140.
+
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. ''The Journal of the Learning Sciences, 4'', 167-207.
 +
 
 +
Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. ''User Modeling and User-Adapted Interaction, 4'', 253-278.
 +
 
 +
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. ''International Journal of Artificial Intelligence in Education, 8'', 30-43.
 +
 
 +
Paas, F., & van Merriënboer, J.J.G. (1993). The efficiency of instructional conditions: An approach to combine mental-effort and performance measures. ''Human Factors, 35'', 737-743.
  
−
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. ''Cognitive Science, 13'', 145-182. [http://rt4rf9qn2y.scholar.serialssolutions.com/?sid=google&auinit=MTH&aulast=CHI&atitle=Self-explanations:+how+students+study+and+use+examples+in+learning+to+solve+problems&title=Cognitive+science&volume=13&issue=2&date=1989&spage=145&issn=0364-0213]
+
Renkl, A., & Atkinson, R. K. (in press). Cognitive skill acquisition: Ordering instructional events in example-based learning. F. E. Ritter, J. Nerb, E. Lehtinen, T. O’Shea (Eds.), ''In order to learn: How ordering effect in machine learning illuminate human learning and vice versa''. Oxford, UK: Oxford University Press.
  
−
Hausmann, R. G. M., & Chi, M. T. H. (2002). Can a computer interface support self-explaining? ''Cognitive Technology, 7''(1), 4-14. [http://www.pitt.edu/~bobhaus/hausmann2002.pdf]
+
Sweller, J., Merriënboer, J. J. G. van, & Paas, F. G. (1998). Cognitive architecture and instructional design. ''Educational Psychology Review, 10'', 251-296.

Latest revision as of 15:38, 31 August 2011

Does learning from worked-out examples improve tutored problem solving?

Alexander Renkl, Rolf Schwonke, Vincent Aleven, & Ron Salden

Abstract

In our research we combine two educational research branches. The first concerns the Cognitive Tutor research branch which deals with offering students the possibility to attain and extend their skills through means of a computerized curriculum of problems. The second branch concerns the research done on worked-out examples and the fading of worked-out steps. Because students’ prior knowledge on novice elements is usually low it is widely believed presenting a worked-out example might give the student more grasp on the novice elements. By presenting more information about the novice elements the student will not have to bridge the gap between his prior knowledge and the novice elements entirely by himself.

However, as the students progresses through a curriculum his knowledge will grow and fully worked-out examples might give him redundant information. Hence, as the curriculum advances the number of worked-out steps in a problem is gradually decreased until all steps are left blank (i.e., problem solving). By continually fading the worked-out steps the student will have more optimal learning instances than either from only fully worked-out problems or only problem solving in the curriculum. Our studies address the main hypothesis which states that an example enriched Cognitive Tutor can create more deep conceptual understanding.

In our recent In Vivo Study (Study 4) we compared a control condition with two experimental conditions which faded worked-out examples. In the control condition the students had to solve problems and enter explanations for each step as they are used to do. In the fixed fading condition the same preset fading of worked-out steps was applied to all students. In the adaptive fading condition the fading of the worked-out steps was based on the individual student's progress as measures on their problem solving and their reasoning. The data of this study have been collected and are currently being processed to the Datashop after which data analysis will commence.

Background and Significance

The background of this research is twofold. (1) The very successful approach of Cognitive Tutors (Anderson, Corbett, Koedinger, & Pelletier, 1995; Koedinger, Anderson, Hadley, & Mark, 1997) is taken up. These computer-based tutors provide individualized support for learning by doing (i.e., solving problems) by selecting appropriate problems to-be-solved, by providing feedback and problem-solving hints, and by on-line assessment of the student’s learning progress. Cognitive Tutors individualize the instruction by selecting problems based on a model of the students’ present knowledge state that is constantly updated, through a Bayesian process called “knowledge tracing” (Corbett & Anderson, 1995). Although problem solving supported by cognitive tutors has been shown to be successful in fostering initial acquisition of cognitive skill, this approach does not seem to be optimal with respect to focusing the learner on the domain principles to be learned. (2) The research tradition on worked-out examples rooted in Cognitive Load Theory (Sweller, van Merriënboer, & Paas, 1998) and, more specifically, the instructional model of example-based learning by Renkl and Atkinson (in press) are taken up in order to foster skill acquisition that is found in deep conceptual understanding. By presenting examples instead of problems to be solved in the beginning of a learning sequence, the learner have more attentional capacity available in order to self-explain and thus deepen their understanding of problem solutions.

In order to foster a deep understanding of domain principles and how they are applied in problem solving, we combine the theoretical rationales of Cognitive Tutors and example-based learning.

Especially, we address the following main hypotheses:

  1. Enriching a Cognitive Tutor unit with examples whose worked-out steps are gradually faded leads to better learning.
  2. Individualizing the fading procedure based on the quality of self-explanations that the learners provide further improves learning.
  3. Using free-form self-explanations is more useful in this context as compared to the usual menu-based formats.
  4. Learning can be enhanced further by providing previously self-explained examples – including the learner’s own self-explanations – as support at problem-solving impasses.

This project is in several respects of significance:

(1) Presently, the positive effects of examples were shown in comparison to unsupported problem solving. We aim to show that example study is also superior to supported problem solving in the very beginning of a learning sequence.

(2) The Cognitive Tutor approach can be enhanced by ideas from research on example-based learning.

(3) The example-based learning approach can be enriched by individualizing instructional procedures such as fading.

Glossary

Salden Learning-from-Examples Glossary

Learning by worked-out examples

Learning by problem solving

Self-explanation

Fading

Research question

Can the effectiveness and efficiency of Cogntive Tutors be enhanced by including learning from worked-out examples?

Independent variables

The independent variable refers to the following variation:

(a) Cognitive Tutor with problems to be solved

versus

(b) Cognitive Tutors with intially worked-out examples, then partially worked-out examples, and finally problem to be solved.

(b1) The fading occurs according to a fixed procedure which is used for all students.

(b2) The fading occurs adaptively based on the knowledge and reason tracing of the individual student's progress.

(c) Previously performed problems are accessible as hints. Meaning that the regular textual hints are replaced by screenshots of the previously performed problems.

Although self-explanation prompts are a typical "ingredient" of example-based learning, but not of learning by problem solving, such prompts were included in all conditions of our studies. Thereby, the potential effects can be clearly attributed to the presence or absense of example study. The exception to this is Study 6 which includes two Untutored conditions. For more information on this study please scroll down to the Further Information section.

Dependent variables

1) Normal post-test of procedural knowledge: measured by the students' performance on new problems dealing with the content they learned in the Cognitive Tutor.

1) Transfer test assessing conceptual knowledge: measured by a variety of questions on the post test that are different in format, non-isomorphic, from those used in training. These include drawing problems, multiple questions and open questions concerning the specific geometric principles the students were exposed to in the Cognitive Tutor.

3) Acceleration of future learning (in future experiments): we plan to use a related unit to the angles and/or circles units which follows each of these units in the curriculum.

4) Learning time: measured in total time to complete the unit in the Cognitive Tutor.

5) Efficiency of learning (relating learning time to learning outcomes): based on the mental efficiency measure developed by Paas and van Merriënboer (1993).

Hypotheses & Results

The provision of example in Cognitive Tutors should lead to better conceptual understanding and, thereby, transfer performance. In addition, examples in Cognitive Tutors should reduce learning time.

On the whole, the present results confirm the hypotheses with respect to conceptual knowledge and learning time. The expected effects on transfer were not found.

Study 1 (lab study at Freiburg, Geometry Cognitive Tutor)

  • Summary
    • Lab Study: 8th grade and 9th grade students from a German high school in Freiburg
    • Domain: translated Circles Unit in the Geometry Cognitive Tutor
    • Start Date: March 1, 2006
    • End Date: March 31, 2006
    • Number of Students: 50
    • Participant Hours: 1.5
    • Data in Datashop: Yes
  • The students were randomly assigned to one of two conditions:
    • Problem Solving Condition: In this control condition students solved answer steps and entered explanations on all problems

Freiburg Summer 2006 - problem solving versus fixed fading examples study - Problem condition-small.jpg

    • Worked Example Condition: In this experimental condition students were first presented with problems that had worked out (i.e., filled in) answer steps but still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.

Freiburg Summer 2006 - problem solving versus fixed fading examples study - Example condition-small.jpg

  • Findings
    • No overall effect of experimental condition on students' conceptual and procedural knowledge on the post-test: t < 1.
    • However, about the same learning outcomes were achieved in shorter learning times in the example-enriched Cognitive Tutor: t(48) = -3.11, p < .001 (one-tailed), r = .41.
    • Accordingly, the efficiency of learning was superior in this latter learning condition: t(48) = 1.73, p < .05 (one-tailed), r = .24 for conceptual knowledge, and t(48) = 1.82, p < .05 (one-tailed), r = .25 for the acquisition of transferable knowledge.

Study 2 (lab study at Freiburg, Geometry Cognitive Tutor)

  • Summary
    • Lab Study: 9th grade and 10th grade students from a German high school in Freiburg
    • Domain: translated Circles Unit in the Geometry Cognitive Tutor
    • Start Date: April 1, 2006
    • End Date: April 31, 2006
    • Number of Students: 30
    • Participant Hours: 1.5
    • Data in Datashop: Yes
  • The students were randomly assigned to one of two conditions:
    • Problem Solving Condition: In this control condition, students solved answer steps and entered explanations on all problems. See Study 1 for a screenshot of this condition.
    • Worked Example Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations. See Study 1 for a screenshot of this condition.
  • Findings
    • With regard to conceptual understanding part of the transfer post-test, an advantage of the example condition over the problem condition was found: t(28) = 1.85, p < .05 (one-tailed), r = 0.33.
    • There were no significant differences on other, procedural, transfer items: t < 1.
    • Similar to Study 1, students in the example condition spent significantly less time than students in the problem solving condition: t(28) = -3.14, p < .001 (one-tailed), r = 0.51.
    • Hence, on a measure of robust learning efficient (i.e., relating performance in terms of the acquisition of conceptual knowledge to the effort in terms of time on task) a large effect was obtained: r = 0.55, t(28) = 3.48, p < .001.

Study 3 (lab study at CMU, Geometry Cognitive Tutor)

  • Summary
    • Lab Study: 8th grade and 9th grade students in rural Pennsylvannia schools
    • Domain: Circles Unit in the Geometry Cognitive Tutor
    • Start Date: October 1, 2006
    • End Date: November 30, 2006
    • Number of Students: 45
    • Participant Hours: 2
    • Data in Datashop: Yes
  • The students were randomly assigned to one of two conditions:
    • Problem Solving Condition: In this control condition, students solved answer steps and entered explanations on all problems. See Study 1 for a screenshot of this condition.
    • Worked Example Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations. See Study 1 for a screenshot of this condition.
  • Findings
    • No overall effect of experimental condition on students' conceptual and procedural knowledge on the post-test: t < 1.
    • No difference in time to complete the Circles Unit in the Tutor: t < 1.

Study 4 (In Vivo study at CMU, Geometry Cognitive Tutor)

  • Summary
    • In Vivo Study: 10th grade geometry classes in rural Pennsylvannia high school
    • Domain: Angles Unit in the Geometry Cognitive Tutor
    • Start Date: January 8, 2007
    • End Date: March 9, 2007
    • Number of Students: 51
    • Participant Hours: 5
    • Data in Datashop: Yes
  • The students were randomly assigned to one of three conditions:
    • Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems.

January 2007 - adaptive fading study CWCTC - Problem solving condition-small.jpg

    • Fixed Fading of Worked Examples Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.

January 2007 - adaptive fading study CWCTC - Fixed fading condition-small.jpg

    • Adaptive Fading of Worked Examples Condition: This experimental condition is similar to the Fixed Fading condition but differs in the fact that the fading of the filled in answer steps is based on the individual student's performance on both the answer and the reason steps.

See screenshot of the Fixed Fading condition, the interface was identical in both condition.

  • Findings
    • Positive effect of the Adaptive Fading Condition over the Problem Solving Condition on the delayed post-test: t(20) = 2.15, p < .05, d = .96.
    • No difference in time to complete the Circles Unit in the Tutor: t < 1.

Study 5 (lab study at Freiburg, Geometry Cognitive Tutor)

  • Summary
    • Lab Study: 9th grade and 10th grade students from a German high school in Freiburg
    • Domain: translated Circles Unit in the Geometry Cognitive Tutor
    • Start Date: March 19, 2007
    • End Date: March 26, 2007
    • Number of Students: 57
    • Participant Hours: 2
    • Data in Datashop: Yes
  • The students were randomly assigned to one of three conditions:
    • Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems.
    • Fixed Fading of Worked Examples Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.
    • Adaptive Fading of Worked Examples Condition: This experimental condition is similar to the Fixed Fading condition but differs in the fact that the fading of the filled in answer steps is based on the individual student's performance on both the answer and the reason steps.
  • Findings
    • A planned contrast comparing the adaptive fading condition with the problem solving + fixed fading conditions revealed higher transfer performance for the adaptive fading condition on the regular post-test: F(1, 54) = 5.05, p < .05, η² = .09. This effect was replicated on the delayed post-test: F(1, 54) = 4.42, p < .05, η² = .08.
    • There were no differences in time spent on either of the post-tests: Fs < 1.

Study 6 (lab study at CMU, Geometry Cognitive Tutor)

  • Summary
    • Lab Study: 8th, 9th and 10th grade students in rural Pennsylvannia school
    • Domain: Circles Unit in the Geometry Cognitive Tutor
    • Start Date: April 18, 2007
    • End Date: April 19, 2007
    • Number of Students: 62
    • Participant Hours: 2
    • Data in Datashop: Yes
  • The students were randomly assigned to one of 4 conditions:
    • Tutored Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems in the standard Cognitive Tutor.
    • Tutored Worked Examples Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.
    • Untutored Problem Solving Condition: In this experimental condition students solved answer steps and entered explanations on all problems in a stripped down version of the Cognitive Tutor. Students could NOT ask for hints and only after entering all answer and reason steps could they receive corrective feedback. They could still browse the Glossary while working on a problem.

Screenshot-notutoring problem solving.jpg

    • Untutored Worked ExamplesProblem Solving Condition: In this experimental condition, students were first presented with problems that had worked-out (i.e., filled in) answer steps, but they still had to enter the explanations for these steps. As they progressed through the Unit, these worked-out answer steps were faded meaning that, towards the end of the Unit, the students had to fill in answer steps and explanations.

Furthermore, students were working in a stripped down version of the Cognitive Tutor. Students could NOT ask for hints and only after entering all answer and reason steps could they receive corrective feedback. They could still browse the Glossary while working on a problem. Screenshot-notutoring worked examples.jpg

  • Findings
    • No overall effect of experimental condition on students' conceptual and procedural knowledge on the post-test: t < 1.
    • No difference in time to complete the Circles Unit in the Tutor: t < 1.

Study 7 (In Vivo study at CMU, Geometry Cognitive Tutor)

  • Summary
    • In Vivo Study: 10th grade geometry classes in two rural Pennsylvannia high schools
    • Domain: Circles Unit in the Geometry Cognitive Tutor
    • Start Date: April 30, 2007
    • End Date: June 1, 2007
    • Number of Students: 104
    • Participant Hours: 5
    • Data in Datashop: Yes
  • The students were randomly assigned to one of 4 conditions:
    • Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems in the standard Cognitive Tutor and received step-by-step textual hints. For a screenshot see the Problem Solving Condition of Study 1.
    • Worked Examples in Hints Condition: In this experimental condition, students solved answer steps and entered explanations on all problems in the standard Cognitive Tutor and for each Geometry concept received a worked example as a hint. In contrast to the control condition which had several hints levels for each step, this experimental condition had only one hint level: the worked example.

Spring 2007 - examples in hints study CWCTC and Wilkinsburg - Examples condition-small.jpg

  • Findings
    • No overall effect of experimental condition on students' conceptual and procedural knowledge on the post-test: t < 1.
    • No difference in time to complete the Circles Unit in the Tutor: t < 1.


Study 8 (In Vivo study at CMU, Geometry Cognitive Tutor)

  • Summary
    • In Vivo Study: 10th grade geometry classes in three rural Pennsylvannia high schools
    • Domain: Circles Unit in the Geometry Cognitive Tutor
    • Start Date: April 15, 2009
    • End Date: June 10, 2009
    • Number of Students: 151
    • Participant Hours: 9-10
    • Data in Datashop: No
  • The students were randomly assigned to one of 3 conditions:
    • Problem Solving Condition: In this control, condition students solved answer steps and entered explanations on all problems in the standard Cognitive Tutor.
    • All Examples Condition: In this experimental condition, students receive worked-out answer steps for all the required problems. There is no fading of these steps thus the first time students get to solve answer steps is on the remedial problem set.
    • Adaptive Fading of Worked Examples Condition: This experimental condition is similar to the All Examples condition but differs in the fact that fading of the filled in answer steps occurs and the fading is based on the individual student's performance on both the answer and the reason steps.


  • Findings
    • The study is still in progress.


Summary of Findings and Explanation

This project belongs to the interactive communication cluster because it investigates a variation of the amount of contribution from the system and from the learner, respectively: Who provides the solution of the initial solution steps?

More specifically, this study is about changes in path choices that occur when a tutoring system includes partially worked examples. The basic idea is that when a tutor relieves a student of most of the work in generating a line by providing part of it, then students are more likely to engage in deep learning to fill in the rest. However, the instruction must be engineered so that students still become autonomous problem solvers—they eventually can do all the work themselves.

In the first German laboratory study, the standard Cognitive Tutor was compared with an example-enriched Cognitive Tutor. While no effects on procedural and conceptual knowledge transfer items were found, the students working with the example-enriched Tutor completed the curriculum faster than the students in the standard Cognitive Tutor. Using the learning time to measure the condition efficiency showed that the example-enriched Tutor obtained higher learning efficiency on the transfer test. Since the German students were inexperienced with the Cognitive Tutor, more detailed instructions were provided in the follow up study. Consequently, the students working on the example-enriched Tutor showed a higher gain on the conceptual knowledge items of the transfer test than the students working with the standard Tutor. Furthermore, similar to the first study the example-enriched Tutor led to significantly shorter learning time than the standard Cognitive Tutor. Lastly, using the learning time to measure efficiency revealed higher learning efficiency for the example-enriched Tutor on the conceptual knowledge items of the transfer test. In terms of the robust learning framework, these results shows that worked-out examples lead to same level of foundational skills in less time. Furthermore, the second study shows that fading worked-out examples can improve sense-making which consequently leads to better robust learning.


Annotated bibliography

Salden R. J. C. M., Aleven, V., Renkl, A., & Wittwer, J. (2006). Does Learning from Examples Improve Tutored Problem Solving? In 2006 Proceedings of the 28th Annual Meeting of the Cognitive Science Society (pp. 2602), Vancouver, Canada. Link to paper

Presentation to the PSLC Advisory Board, Fall 2006.

Schwonke, R., Wittwer, J., Aleven, V., Salden, R. J. C. M., Krieg, C., & Renkl, A. (2007). Can tutored problem solving benefit from faded worked-out examples? Paper presented at The European Cognitive Science Conference 2007, May 23-27. Delphi, Greece. Link to paper

Salden, R., Aleven, V., & Renkl, A. (2007). Can tutored problem solving be improved by learning from examples? Proceedings of the 29th Annual Conference of the Cognitive Science Society (p. 1847), Nashville, USA.

Salden, R., Aleven, V., & Renkl, A., & Schwonke, R. (2008). Worked Examples and the Assistance Dilemma. Paper presented at The American Educational Research Association 2008, March 23-27. New York, USA. Link to paper

Salden, R., Aleven, V., Schwonke, R., & Renkl, A. (2008, June). Worked Are Worked Examples and Tutored Problem Solving Synergistic Forms of Support? Poster presented at the 8th International Conference of the Learning Sciences (ICLS). Link to paper

Salden, R., Aleven, V., & Renkl, A., & Schwonke, R. (2008, July). Worked Examples and Tutored Problem Solving: Redundant or Synergistic Forms of Support? Paper presented at the 30th Annual Meeting of the Cognitive Science Society, July 23-26. Washington DC, USA. Winner of the "Cognition and Student Learning" award

Schwonke, R., Renkel, A., Krieg, C, Wittwer, J., Aleven, V., Salden, R. J. C. M. (2009). The Worked-example Effect: Not an Artefact of Lousy Control Conditions. Computers in Human Behavior, 25, 258-266.

Salden, R. J. C. M., Aleven, V. A. W. M. M., Renkl, A., & Schwonke, R. (2009). Worked examples and tutored problem solving: Redundant or synergistic forms of support? Topics in Cognitive Science, 1, 203-213.

Salden, R. J. C. M., Aleven, V. A. W. M. M., Schwonke, R., & Renkl, A. (2009). The expertise reversal effect and worked examples in tutored problem solving. Manuscript submitted for publication.


References

Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4, 167-207.

Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.

Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.

Paas, F., & van Merriënboer, J.J.G. (1993). The efficiency of instructional conditions: An approach to combine mental-effort and performance measures. Human Factors, 35, 737-743.

Renkl, A., & Atkinson, R. K. (in press). Cognitive skill acquisition: Ordering instructional events in example-based learning. F. E. Ritter, J. Nerb, E. Lehtinen, T. O’Shea (Eds.), In order to learn: How ordering effect in machine learning illuminate human learning and vice versa. Oxford, UK: Oxford University Press.

Sweller, J., Merriënboer, J. J. G. van, & Paas, F. G. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251-296.

Retrieved from "https://learnlab.org/wiki/index.php?title=Does_learning_from_worked-out_examples_improve_tutored_problem_solving%3F&oldid=12183"