Reflective Dialogues (Katz)

Do Reflective Dialogues that Explicitly Target the “What? How? and Why (not)?” Knowledge of Physics Problem Solving Promote Expert-like Planning Ability?

Sandra Katz

Summary Table

PIs	Sandra Katz, John Connelly, Donald Treacy
Study Start Date	3/1/06
Study End Date	6/30/07
LearnLab Site	USNA
LearnLab Course	General Physics I
Number of Students	N = 67
Total Participant Hours	approx. 750 hrs
DataShop	Yes

Abstract

One of the main differences between experts and novices in physics is that experts are more adept at identifying relevant principles and generating solution plans before starting to solve a problem (Chi, Glaser & Rees, 1982; Dufresne, Gerace, Hardiman, & Mestre, 1992; Priest & Lindsay, 1992). We propose that this difference may be due to two reasons: (1) in traditional physics courses, students are not explicitly asked to plan or are given scaffolding to support planning during problem solving, and (2) many students lack the basic knowledge of physics concepts, principles, and procedures that is prerequisite for effective planning.

In this project, we tested the effectiveness of engaging students in reflective dialogues after they solve problems in Andes that explicitly target the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why (or why not) to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). The main hypothesis tested is that explicit training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, will be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

To test this hypothesis, we conducted a Physics LearnLab study in which we compared the effectiveness of an experimental version of Andes that engaged students in reflective “what? how? and why?” dialogues after physics problem solving (explicit instruction condition) with a control version that gave students additional problem-solving practice after they solved an Andes problem—e.g., by having them identify bugs in a sample student solution to a problem (implicit instruction condition). We predicted that students in the explicit condition would outperform students in the implicit condition with respect to gain score from pre-test to post-test and scores on course exams that targeted far transfer. Consistent with prior research, the dialogues promoted conceptual understanding of physics. However, measures of retention and transfer to problem-solving ability showed only a marginal effect of completing the reflective dialogues.

Glossary

• Post-practice reflection
• Reflection questions
• Knowledge Construction Dialogues (KCDs)

Research question

Does explicit training in the three main components of problem-solving knowledge—i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them—enhance students’ problem-solving ability?

Background and Significance

Because experts develop schemata largely through a good deal of practice with solving various types of problems, traditional introductory college physics courses take a similar approach. They are structured so that students are introduced to concepts, principles, and procedures in their text and through lectures, which they are then asked to apply to many problem-solving exercises. The hope is that, with extensive practice, students will integrate conceptual and procedural knowledge and develop expert-like schemata and planning skills. Unfortunately, many students don’t. In addition to exiting these courses with lingering naïve misconceptions (e.g., Halloun & Hestenes, 1985; McDermott, 1984), they continue to solve problems largely by manipulating equations until the desired quantity is isolated—that is, by means-end analysis—instead of by identifying relevant principles and generating solution plans before starting to solve a problem, as experts do (Dufresne et al., 1992; Larkin, McDermott, Simon, & Simon, 1980; Priest & Lindsay, 1992).

We refer to the traditional approach to physics instruction described in the preceding paragraph as implicit instruction, because it encourages the inductive development of abstractions (concepts, principles, and schemata) through repeated exposure to instances, instead of by explicitly reifying these abstractions (O'Malley & Chamot, 1994). In response to the limitations of implicit approaches to physics instruction that neither ask students to plan nor scaffold them in doing it, several instructional scientists have proposed methods that explicitly engage students in planning exercises (Dufresne et al., 1992; Leonard, Dufresne, & Mestre, 1996; Mestre, Dufresne, Gerace, & Hardiman, 1993). These methods have met with modest success when tested mainly using high-achieving students (B or above in an introductory college physics course). We suggest that the main reason that many students, even high achievers, are unable to use explicit planning methods effectively is that they lack the basic concepts, principles, and procedures that are prerequisites for effective planning.

This project tested the effectiveness of engaging students in reflective dialogues after they solve problems in Andes (e.g., Gertner & VanLehn, 2000) which explicitly targeted the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). Explicit instruction on these knowledge components proved to be effective and more efficient than giving students repeated practice in solving different types of problems.

Dependent Variables

• Gains in qualitative and quantitative knowledge. Post-test score,and pre-test to post-test gain scores, on near and far transfer items.
• Short-term retention. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
• Long-term retention. Performance on final exam, taken several weeks after the intervention.

Independent Variables

Students within each course section were block-randomly assigned to one of two dialogue conditions: standard Knowledge Construction Dialogues (KCDs) or control with no KCDs. Students in the control condition were assigned an additional five Andes problems to attempt to better equate time on task.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

• Number of problems completed before the post-test was administered
• Number of dialogues that the student completed
• Grade point average (CQPR)
• College major
• Pre-test score

Hypothesis

The main hypothesis tested was that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

Findings

Data analyses are still in progress, but below are our findings to date:

Analyses of pre- and post-test scores were more encouraging than for last year's study (Post-practice reflection (Katz)). After omitting scores from one student with a post-test duration of less than two minutes, we were left with treatment and control groups of equal size (n = 33). We also re-classified two treatment subjects who did no dialogues as control subjects (effective treatment and control ns = 31 and 35, respectively), although these subjects were not able to access the five extra control-group Andes problems. As we had hoped, ANOVA showed no significant differences between the effective treatment and control groups on pre-test score (respective Ms = 12.10 and 11.77; F < 1), but treatment subjects had higher mean post-test scores (17.97 vs. 15.57; F(1, 64) = 4.89, p = .031), mean raw gain scores (5.87 vs. 3.80; F = 5.62, p = .021), and mean Estes gain scores (0.330 vs. 0.208; F = 6.74, p = .012) than did control subjects. In short, without regard to problem-solving practice, subjects who did KCDs did significantly better on the post-test than those who did not.

Student participation in both homework problem solving and dialogues was much improved over last year, with 21 of 34 control subjects (62%) and 23 of 33 treatment subjects (70%) finishing at least 80% of the assigned target problems (of 31 for control, 26 for treatment) prior to the post-test and with 25 treatment subjects (76%) finishing at least 80% of the 26 associated KCDs. However, participation was still far from perfect; 7 treatment subjects (21%) completed half or fewer of the assigned KCDs on time, and 2 of them completed no target problems or KCDs on time. We again treated KCD completion and target problem completion as continuous independent variables in regression analyses. Regressing post-test score on pre-test score, QPA, number of KCDs completed, and number of target problems completed (R² = .52, F(4, 61) = 16.70, p < .00001) showed positive contributions of all factors, but only pre-test score, QPA, and KCD completion were statistically significant (ps < .001, .05, & .05, respectively); problem completion was ns (p = .54). Therefore, across all subjects it was KCD completion, as opposed to target homework problem completion, that significantly predicted post-test performance.

To measure retention and transfer, we also analyzed scores from an hourly exam administered by one instructor to his two sections (n=47). All problems on this exam were quantitative and covered a subset of the units targeted during the intervention. Scores on this exam ranged from 176 to 382 (of 400), and were highly correlated with pre- and post-test scores; rs(45) = .54 and .71, ps < .0001 and .00001. However, ANOVAs showed no differences between groups (Fs < 1), and regression of subscores on QPA, KCDs completed, and target problems completed (R² = .54, F(3, 43) = 16.58, p < .00001) showed positive contributions of all factors but a significant effect of only QPA (p < .00001). Therefore, student performance on the hourly exam was not significantly affected by either KCD completion or target problem completion.

In summary, student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed prior to the post-test. However, neither measure had a significant effect on student performance on an exam that focused on problem-solving ability.

Explanation

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

• The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
• The student takes the learning-by-doing path unless it becomes too difficult. More students took the learning-by doing path than in our prior study, perhaps because this year instructors required them to do so (i.e., this time there were negative consequences for avoiding them).

Analyses of retention and transfer to problem-solving ability did not reveal a significant effect of the reflective dialogues. We had hoped that the capstone dialogues that treatment students completed before selected problems, at the end of each unit, would support problem-solving ability. It is possible that the intervention consisted of too few capstone dialogues (only five) to observe any effect.

What seems to be the critical feature of post-practice reflection (PPR) in supporting conceptual understanding is the explicit instruction that it provides in domain knowledge. PPR may help students to fill in knowledge gaps, resolve misconceptions, and abstract from the case at hand so that they are better prepared to engage in constructive activity (e.g., self-explanation) in future problems. Preliminary research comparing Andes (which encourages implicit learning of problem-solving strategies) with Pyrenees, a system that teaches problem-solving strategies explicitly, also suggests an advantage for explicit instruction (VanLehn et al., 2004). Further research is needed to identify the mechanisms that drive learning from reflective dialogue, and to increase its potential to enhance problem-solving ability in addition to conceptual knowledge.

Further Information

Annotated bibliography

• Presentation to site visitors, 2005
• Full paper accepted at AIED 2007:
  • Katz, S., Connelly, J., & Wilson, C. (in press). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. Proceedings of AIED07.

References

• Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), Advances in the Psychology of Human Intelligence, Vol. 1 (pp. 7-75). Hillsdale, NJ: Erlbaum.
• Dufresne, R. J., Gerace, P., Hardiman, T., & Mestre, J. P. (1992). Constraining novices to perform expertlike analyses: Effects on schema acquisition. The Journal of the Learning Sciences, 2 (3), 307-331.
• Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), ITS 2000: Proceedings of the 5th International Conference on Intelligent Tutoring Systems (pp. 133-142). Berlin: Springer-Verlag.
• Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208, 1335-1342.
• Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. American Journal of Physics, 64 (12), 1495-1503.
• Mestre, J. P., Dufresne, R. J., Gerace, W. J., & Hardiman, P. T. (1993). Promoting skilled problem-solving behavior among beginning physics students. Journal of Research in Science Teaching, 30, 303-317.
• O’Malley, M., and Chamot, A. (1994). The CALLA Handbook. Reading, MA: Addison-Wesley.
• Priest, A. G., & Lindsay, R. O. (1992). New light on novice-expert differences in physics problem solving. British Journal of Psychology, 83, 389-405.
• VanLehn, K., Bhembe, D., Chi, M., Lynch, C., Schulze, K., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2004). Implicit versus explicit learning of strategies in a nonprocedural cognitive skill. In J. C. Lester, R. M. Vicari, & F. Paraguacu, (Eds.), Intelligent Tutoring Systems: 7th International Conference (pp. 521-530). Berlin: Springer-Verlag.

Connections

This project shares features with the following research projects:

Use of Questions during learning

• Post-practice reflection (Katz)
• Deep-level questions during example studying (Craig & Chi)
• FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)
• Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)

Self explanations during learning

• Deep-level questions during example studying (Craig & Chi)
• The Effects of Interaction on Robust Learning (Hausmann & Chi)
• A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)

Future plans

Our future plans for June 2007 - December 2007:

• write journal article expanding AIED conference paper