Extending Reflective Dialogue Support (Katz & Connelly)

From LearnLab
Revision as of 22:09, 22 December 2008 by Connelly (talk | contribs) (→‎Dependent Variables)
Jump to: navigation, search

Extending Automated Dialogue Support for Robust Learning of Physics

Sandra Katz

Summary Table

PIs Sandra Katz & John Connelly
Study Start Date 10/1/07
Study End Date 9/30/08
LearnLab Site USNA
LearnLab Course General Physics I
Number of Students N = 75
Total Participant Hours approx. 125 hrs
DataShop Yes

Abstract

Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to integrate quantitative and qualitative knowledge via post-practice reflection dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study (Katz, Connelly, & Treacy) by trying to better support robust learning via a third condition in which students work through longer dialogues designed to foster transfer by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

Glossary

Research question

Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these knowledge components in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster transfer and robust learning?

Background and Significance

Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to integrate quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, integrative reflective dialogues that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the Andes physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study (Katz, Connelly, & Treacy), this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more robust learning measures of transfer (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (what if scenarios), designed to support both qualitative and quantitative knowledge (most of our previous what if scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context and to new contexts, to help support near and far transfer; and (3) students were prompted to state the rules (knowledge components) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

Dependent Variables

  • Gains in qualitative and quantitative knowledge. Post-test score, and pre-test to post-test gain scores, on near and far transfer items.
  • Long-term retention. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure short-term retention via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

Independent Variables

Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of Knowledge Construction Dialogues (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study (Katz, Connelly, & Treacy), in which mostly conceptual dialogues followed problem solving on selected Andes problems.

6kcd.jpg

The new long-KCD condition appended more quantitative practice and transfer content (via additional what if scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

7lkcd.jpg

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

E1b-50.jpg

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

  • Number of problems completed before the post-test was administered
  • Number of dialogues that the student completed
  • Grade point average (CQPR)
  • College major category
  • Pre-test score

Hypothesis

The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these knowledge components in different problem-solving contexts would better foster transfer and robust learning.

Findings

Preliminary analyses show at least marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of general "gaming the system" by some students, more detailed analyses are pending the identification and omission of noisy data from our overall corpus.

Further Information

Annotated bibliography

  • Presentation to Advisory Board, January 2008

References

  • Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work (pp. 425-432). Amsterdam: IOS Press.
  • Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. American Journal of Physics, 64 (12), 1495-1503.
  • VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3).
  • VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), Artificial Intelligence in Education (pp. 678-685). Amsterdam, Netherlands: IOS Press.

Connections

This project shares features with the following research projects:

Use of Questions during learning

Self explanations during learning

Future plans

Our future plans through September 2008:

  • continue to try extracting viable data
  • write journal article reporting updated findings
Retrieved from "https://learnlab.org/wiki/index.php?title=Extending_Reflective_Dialogue_Support_(Katz_%26_Connelly)&oldid=8721"