Reflective Dialogues (Katz)

Do Reflective Dialogues that Explicitly Target the “What? How? and Why (not)?” Knowledge of Physics Problem Solving Promote Expert-like Planning Ability?

Sandra Katz

Summary Table

PIs	Sandra Katz, John Connelly, Donald Treacy
Study Start Date	3/1/06
Study End Date	6/30/07
LearnLab Site	USNA
LearnLab Course	General Physics I
Number of Students	N = 67
Total Participant Hours	approx. 750 hrs
DataShop	No; Andes data still incompatible

Abstract

One of the main differences between experts and novices in physics is that experts are more adept at identifying relevant principles and generating solution plans before starting to solve a problem (Dufresne, Gerace, Hardiman, & Mestre, 1992; Chi, Glaser & Rees, 1982; Priest & Lindsay, 1992). We propose that this difference may be due to two reasons: (1) in traditional physics courses, students are not explicitly asked to plan or are they given scaffolding to support planning during problem solving, and (2) many students lack the basic knowledge of physics concepts, principles, and procedures that is prerequisite for effective planning.

In this project, we are testing the effectiveness of engaging students in reflective dialogues after they solve problems in the Andes that explicitly target the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why (or why not) to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). The main hypothesis to be tested is that explicit training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, will be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

To test this hypothesis, we will conduct a Physics LearnLab study in which we compare the effectiveness of an experimental version of Andes that engages students in reflective “what? how? and why?” dialogues after physics problem solving (explicit instruction condition) with a control version that gives students additional problem-solving practice after they solve an Andes problem—e.g., by having them identify bugs in a sample student solution to a problem (implicit instruction condition). We predict that students in the explicit condition will outperform students in the implicit condition with respect to gain score from pre-test to post-test and scores on course exams that target far transfer. The main contribution of this study will be to determine if explicit instruction in planning and in the component knowledge and skills needed for planning via reflective dialogues can promote students’ acquisition of solution schemata.

Glossary

• Post-practice reflection
• Reflection questions
• Knowledge Construction Dialogues (KCDs)

Research question

Does explicit training in the three main components of problem-solving knowledge—i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them—enhance students’ problem-solving ability?

Background and Significance

Because experts develop schemata largely through a good deal of practice with solving various types of problems, traditional introductory college physics courses take a similar approach. They are structured so that students are introduced to concepts, principles, and procedures in their text and through lectures, which they are then asked to apply to many problem-solving exercises. The hope is that, with extensive practice, students will integrate conceptual and procedural knowledge and develop expert-like schemata and planning skills. Unfortunately, many students don’t. In addition to exiting these courses with lingering naïve misconceptions (e.g., Halloun & Hestenes, 1985; McDermott, 1984), they continue to solve problems largely by manipulating equations until the desired quantity is isolated—that is, by means-end analysis—instead of by identifying relevant principles and generating solution plans before starting to solve a problem, as experts do (Dufrene et al., 1992; Larkin, McDermott, Simon, & Simon, 1980; Priest & Lindsay, 1992).

We refer to the traditional approach to physics instruction described in the preceding paragraph as implicit instruction, because it encourages the inductive development of abstractions (concepts, principles, and schemata) through repeated exposure to instances, instead of by explicitly reifying these abstractions (O'Malley & Chamot, 1994). In response to the limitations of implicit approaches to physics instruction that neither ask students to plan nor scaffold them in doing it, several instructional scientists have proposed methods that explicitly engage students in planning exercises (Dufresne et al., 1992; Leonard, Dufresne, & Mestre, 1996; Mestre, Dufresne, Gerace, & Hardiman, 1993). These methods have met with modest success when tested mainly using high-achieving students (B or above in an introductory college physics course). We propose that the main reason that many students, even high achievers, are unable to use explicit planning methods effectively is that they lack the basic concepts, principles, and procedures that are prerequisites for effective planning.

This project tests the effectiveness of engaging students in reflective dialogues after they solve problems in Andes (e.g., Gertner & VanLehn, 2000) which explicitly target the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). If explicit instruction on these knowledge components proves to be effective, we will have identified an instructional strategy that facilitates the acquisition of problem-solving schema—one that is more efficient than repeated practice in solving many problems.

Dependent Variables

• Gains in qualitative and quantitative knowledge. Post-test score,and pre-test to post-test gain scores, on near and far transfer items.
• Short-term retention. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
• Longer-term retention. Performance on final exam, taken several weeks after the intervention.

Independent Variables

Students within each course section were block-randomly assigned to one of two dialogue conditions: standard KCDs (KCD) or control (Ctrl) with no KCDs. Students in Ctrl were assigned an additional five Andes problems to equate time on task.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

• Number of problems completed before the post-test was administered
• Number of dialogues that the student completed
• CQPR—grade point average
• College major
• Pre-test score

Hypothesis

The main hypothesis to be tested is that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, will be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

Findings

Data collection is in progress.

Further Information

Annotated bibliography

• Presentation to site visitors, 2005
• Full-paper accepted at AIED 2007

References

• Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), Advances in the Psychology of Human Intelligence, Vol. 1 (pp. 7-75). Hillsdale, NJ: Erlbaum.
• Dufresne, R. J., Gerace, P., Hardiman, T., & Mestre, J. P. (1992). Constraining novices to perform expertlike analyses: Effects on schema acquisition. The Journal of the Learning Sciences, 2 (3), 307-331.
• Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), ITS 2000: Proceedings of the 5th International Conference on Intelligent Tutoring Systems (pp. 133-142). Berlin: Springer-Verlag.
• Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208, 1335-1342.
• Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. American Journal of Physics, 64 (12), 1495-1503.
• Mestre, J. P., Dufresne, R. J., Gerace, W. J., & Hardiman, P. T. (1993). Promoting skilled problem-solving behavior among beginning physics students. Journal of Research in Science Teaching, 30, 303-317.
• O’Malley, M., and Chamot, A. (1994). The CALLA Handbook. Reading, MA: Addison-Wesley.
• Priest, A. G., & Lindsay, R. O. (1992). New light on novice-expert differences in physics problem solving. British Journal of Psychology, 83, 389-405.

Connections

This project shares features with the following research projects:

Use of Questions during learning

• Post-practice reflection (Katz)
• Deep-level questions during example studying (Craig & Chi)
• FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)

Self explanations during learning

• Deep-level questions during example studying (Craig & Chi)
• The Effects of Interaction on Robust Learning (Hausmann & Chi)
• A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)

Future plans

Our future plans for June 2007 - December 2007:

• Transcribe the interactions and code them for explanations or paraphrases.
• Link codes from transcript to knowledge components.
• Link transcript to steps in Andes log files.
• Submit poster to CogSci 2007 for the learning results