Theory Wiki - User contributions [en]

Extending Reflective Dialogue Support (Katz & Connelly)

2009-05-04T21:07:29Z

Connelly: /* Future plans */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

==== Final Exam & Course Grades ====

A regression of exam score on CQPR, KCD completion counts, and problem completion counts was significant (''R''2 = .47, ''F''(3, 68) = 20.17, ''p'' < .00001) and showed positive contributions of all three factors, but only CQPR had a statistically significant effect (''p'' < .00001). However, when I omitted CQPR from the model (''R''2 = .16, ''F''(2, 72) = 7.08, ''p'' < .005), both KCD and problem completion counts reached statistical significance (''p''s < .05). A regression of final grade on all three factors was significant (''R''2 = .56, ''F''(3, 53) = 22.10, ''p'' < .00001), but again only CQPR had a statistically significant effect (p < .00001). When I omitted CQPR from this model (''R''2 = .13, ''F''(2, 55) = 3.99, ''p'' < .05), neither factor of interest reached significance, but KCD completion had a stronger marginal effect (''p'' = .073) than did problem completion (''p'' = .106).

=== Explanation ===

Despite being unable to perform many of our intended analyses, including examinations of group differences between our old Short KCDs and new Long KCDs, we were able to replicate our prior finding (Katz et al. 2007) that it was KCD completion, as opposed to the completion of target homework problems, that significantly (for most scores and subscores) improved post-test performance. In other words, whether the KCDs were Short or Long, doing more of them better predicted post-test scores than did the solving of homework problems.

Moreover, we found that both factors significantly improved scores on the final exam. That is, the more KCDs and target homework problems students did, the better they performed on the final exam. Although the effects of both factors were not statistically significant on their final course grades, the same trend existed, with KCDs being a stronger predictor of course grades that were homework problems.

This marked the first time in our line of work with KCDs (Connelly & Katz, 2006; Katz et al., 2005, 2007) that learning benefits of our dialogues transferred to longer-term performance measures. Future work could investigate the relative benefits of our Short, largely qualitative KCDs versus our Long KCDs with both qualitative and quantitative knowledge, as well as explicit ties between them.

This study also has implications for other studies of learning gains from instructional interventions, which tend to show small (if any) effects. Investigator attempts to “clean” their data, after being confronted with “cheating” and “gaming” behaviors, may be worthwhile in that they might increase the signal to noise ratio enough for intended (or perhaps even unintended) learning effects to emerge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008
* Virtual brief paper (results with salvaged data) accepted at ED-MEDIA09:
** Connelly, J., & Katz, S. (in press). Toward more robust learning of physics via reflective dialogue extensions. To appear in ''Proceedings of ED-MEDIA 2009''

==== References ====
*Connelly, J., & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition and Learning, 4'', 1-29. Philadelphia, PA: Old City Publishing.
*Katz, S., Connelly, J., & Wilson, C. (2005). When should dialogues in a scaffolded learning environment take place? In P. Kommers & G. Richards (Eds.), ''Proceedings of ED-MEDIA 2005'' (pp. 2850-2855). Norfolk, VA: AACE.
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* resubmit rejected conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2009-05-04T21:07:08Z

Connelly: /* Annotated bibliography */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

==== Final Exam & Course Grades ====

A regression of exam score on CQPR, KCD completion counts, and problem completion counts was significant (''R''2 = .47, ''F''(3, 68) = 20.17, ''p'' < .00001) and showed positive contributions of all three factors, but only CQPR had a statistically significant effect (''p'' < .00001). However, when I omitted CQPR from the model (''R''2 = .16, ''F''(2, 72) = 7.08, ''p'' < .005), both KCD and problem completion counts reached statistical significance (''p''s < .05). A regression of final grade on all three factors was significant (''R''2 = .56, ''F''(3, 53) = 22.10, ''p'' < .00001), but again only CQPR had a statistically significant effect (p < .00001). When I omitted CQPR from this model (''R''2 = .13, ''F''(2, 55) = 3.99, ''p'' < .05), neither factor of interest reached significance, but KCD completion had a stronger marginal effect (''p'' = .073) than did problem completion (''p'' = .106).

=== Explanation ===

Despite being unable to perform many of our intended analyses, including examinations of group differences between our old Short KCDs and new Long KCDs, we were able to replicate our prior finding (Katz et al. 2007) that it was KCD completion, as opposed to the completion of target homework problems, that significantly (for most scores and subscores) improved post-test performance. In other words, whether the KCDs were Short or Long, doing more of them better predicted post-test scores than did the solving of homework problems.

Moreover, we found that both factors significantly improved scores on the final exam. That is, the more KCDs and target homework problems students did, the better they performed on the final exam. Although the effects of both factors were not statistically significant on their final course grades, the same trend existed, with KCDs being a stronger predictor of course grades that were homework problems.

This marked the first time in our line of work with KCDs (Connelly & Katz, 2006; Katz et al., 2005, 2007) that learning benefits of our dialogues transferred to longer-term performance measures. Future work could investigate the relative benefits of our Short, largely qualitative KCDs versus our Long KCDs with both qualitative and quantitative knowledge, as well as explicit ties between them.

This study also has implications for other studies of learning gains from instructional interventions, which tend to show small (if any) effects. Investigator attempts to “clean” their data, after being confronted with “cheating” and “gaming” behaviors, may be worthwhile in that they might increase the signal to noise ratio enough for intended (or perhaps even unintended) learning effects to emerge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008
* Virtual brief paper (results with salvaged data) accepted at ED-MEDIA09:
** Connelly, J., & Katz, S. (in press). Toward more robust learning of physics via reflective dialogue extensions. To appear in ''Proceedings of ED-MEDIA 2009''

==== References ====
*Connelly, J., & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition and Learning, 4'', 1-29. Philadelphia, PA: Old City Publishing.
*Katz, S., Connelly, J., & Wilson, C. (2005). When should dialogues in a scaffolded learning environment take place? In P. Kommers & G. Richards (Eds.), ''Proceedings of ED-MEDIA 2005'' (pp. 2850-2855). Norfolk, VA: AACE.
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2009-05-04T21:06:34Z

Connelly: /* Annotated bibliography */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

==== Final Exam & Course Grades ====

A regression of exam score on CQPR, KCD completion counts, and problem completion counts was significant (''R''2 = .47, ''F''(3, 68) = 20.17, ''p'' < .00001) and showed positive contributions of all three factors, but only CQPR had a statistically significant effect (''p'' < .00001). However, when I omitted CQPR from the model (''R''2 = .16, ''F''(2, 72) = 7.08, ''p'' < .005), both KCD and problem completion counts reached statistical significance (''p''s < .05). A regression of final grade on all three factors was significant (''R''2 = .56, ''F''(3, 53) = 22.10, ''p'' < .00001), but again only CQPR had a statistically significant effect (p < .00001). When I omitted CQPR from this model (''R''2 = .13, ''F''(2, 55) = 3.99, ''p'' < .05), neither factor of interest reached significance, but KCD completion had a stronger marginal effect (''p'' = .073) than did problem completion (''p'' = .106).

=== Explanation ===

Despite being unable to perform many of our intended analyses, including examinations of group differences between our old Short KCDs and new Long KCDs, we were able to replicate our prior finding (Katz et al. 2007) that it was KCD completion, as opposed to the completion of target homework problems, that significantly (for most scores and subscores) improved post-test performance. In other words, whether the KCDs were Short or Long, doing more of them better predicted post-test scores than did the solving of homework problems.

Moreover, we found that both factors significantly improved scores on the final exam. That is, the more KCDs and target homework problems students did, the better they performed on the final exam. Although the effects of both factors were not statistically significant on their final course grades, the same trend existed, with KCDs being a stronger predictor of course grades that were homework problems.

This marked the first time in our line of work with KCDs (Connelly & Katz, 2006; Katz et al., 2005, 2007) that learning benefits of our dialogues transferred to longer-term performance measures. Future work could investigate the relative benefits of our Short, largely qualitative KCDs versus our Long KCDs with both qualitative and quantitative knowledge, as well as explicit ties between them.

This study also has implications for other studies of learning gains from instructional interventions, which tend to show small (if any) effects. Investigator attempts to “clean” their data, after being confronted with “cheating” and “gaming” behaviors, may be worthwhile in that they might increase the signal to noise ratio enough for intended (or perhaps even unintended) learning effects to emerge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008
* Virtual brief paper on salvaged data accepted at ED-MEDIA09:
** Connelly, J., & Katz, S. (in press). Toward more robust learning of physics via reflective dialogue extensions. To appear in ''Proceedings of ED-MEDIA 2009''

==== References ====
*Connelly, J., & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition and Learning, 4'', 1-29. Philadelphia, PA: Old City Publishing.
*Katz, S., Connelly, J., & Wilson, C. (2005). When should dialogues in a scaffolded learning environment take place? In P. Kommers & G. Richards (Eds.), ''Proceedings of ED-MEDIA 2005'' (pp. 2850-2855). Norfolk, VA: AACE.
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2009-05-04T21:05:33Z

Connelly: /* Annotated bibliography */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

==== Final Exam & Course Grades ====

A regression of exam score on CQPR, KCD completion counts, and problem completion counts was significant (''R''2 = .47, ''F''(3, 68) = 20.17, ''p'' < .00001) and showed positive contributions of all three factors, but only CQPR had a statistically significant effect (''p'' < .00001). However, when I omitted CQPR from the model (''R''2 = .16, ''F''(2, 72) = 7.08, ''p'' < .005), both KCD and problem completion counts reached statistical significance (''p''s < .05). A regression of final grade on all three factors was significant (''R''2 = .56, ''F''(3, 53) = 22.10, ''p'' < .00001), but again only CQPR had a statistically significant effect (p < .00001). When I omitted CQPR from this model (''R''2 = .13, ''F''(2, 55) = 3.99, ''p'' < .05), neither factor of interest reached significance, but KCD completion had a stronger marginal effect (''p'' = .073) than did problem completion (''p'' = .106).

=== Explanation ===

Despite being unable to perform many of our intended analyses, including examinations of group differences between our old Short KCDs and new Long KCDs, we were able to replicate our prior finding (Katz et al. 2007) that it was KCD completion, as opposed to the completion of target homework problems, that significantly (for most scores and subscores) improved post-test performance. In other words, whether the KCDs were Short or Long, doing more of them better predicted post-test scores than did the solving of homework problems.

Moreover, we found that both factors significantly improved scores on the final exam. That is, the more KCDs and target homework problems students did, the better they performed on the final exam. Although the effects of both factors were not statistically significant on their final course grades, the same trend existed, with KCDs being a stronger predictor of course grades that were homework problems.

This marked the first time in our line of work with KCDs (Connelly & Katz, 2006; Katz et al., 2005, 2007) that learning benefits of our dialogues transferred to longer-term performance measures. Future work could investigate the relative benefits of our Short, largely qualitative KCDs versus our Long KCDs with both qualitative and quantitative knowledge, as well as explicit ties between them.

This study also has implications for other studies of learning gains from instructional interventions, which tend to show small (if any) effects. Investigator attempts to “clean” their data, after being confronted with “cheating” and “gaming” behaviors, may be worthwhile in that they might increase the signal to noise ratio enough for intended (or perhaps even unintended) learning effects to emerge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008
* Virtual brief paper accepted at ED-MEDIA09:
** Connelly, J., & Katz, S. (in press). Toward more robust learning of physics via reflective dialogue extensions. To appear in ''Proceedings of ED-MEDIA 2009''

==== References ====
*Connelly, J., & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition and Learning, 4'', 1-29. Philadelphia, PA: Old City Publishing.
*Katz, S., Connelly, J., & Wilson, C. (2005). When should dialogues in a scaffolded learning environment take place? In P. Kommers & G. Richards (Eds.), ''Proceedings of ED-MEDIA 2005'' (pp. 2850-2855). Norfolk, VA: AACE.
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:51:35Z

Connelly: /* References */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

==== Final Exam & Course Grades ====

A regression of exam score on CQPR, KCD completion counts, and problem completion counts was significant (''R''2 = .47, ''F''(3, 68) = 20.17, ''p'' < .00001) and showed positive contributions of all three factors, but only CQPR had a statistically significant effect (''p'' < .00001). However, when I omitted CQPR from the model (''R''2 = .16, ''F''(2, 72) = 7.08, ''p'' < .005), both KCD and problem completion counts reached statistical significance (''p''s < .05). A regression of final grade on all three factors was significant (''R''2 = .56, ''F''(3, 53) = 22.10, ''p'' < .00001), but again only CQPR had a statistically significant effect (p < .00001). When I omitted CQPR from this model (''R''2 = .13, ''F''(2, 55) = 3.99, ''p'' < .05), neither factor of interest reached significance, but KCD completion had a stronger marginal effect (''p'' = .073) than did problem completion (''p'' = .106).

=== Explanation ===

Despite being unable to perform many of our intended analyses, including examinations of group differences between our old Short KCDs and new Long KCDs, we were able to replicate our prior finding (Katz et al. 2007) that it was KCD completion, as opposed to the completion of target homework problems, that significantly (for most scores and subscores) improved post-test performance. In other words, whether the KCDs were Short or Long, doing more of them better predicted post-test scores than did the solving of homework problems.

Moreover, we found that both factors significantly improved scores on the final exam. That is, the more KCDs and target homework problems students did, the better they performed on the final exam. Although the effects of both factors were not statistically significant on their final course grades, the same trend existed, with KCDs being a stronger predictor of course grades that were homework problems.

This marked the first time in our line of work with KCDs (Connelly & Katz, 2006; Katz et al., 2005, 2007) that learning benefits of our dialogues transferred to longer-term performance measures. Future work could investigate the relative benefits of our Short, largely qualitative KCDs versus our Long KCDs with both qualitative and quantitative knowledge, as well as explicit ties between them.

This study also has implications for other studies of learning gains from instructional interventions, which tend to show small (if any) effects. Investigator attempts to “clean” their data, after being confronted with “cheating” and “gaming” behaviors, may be worthwhile in that they might increase the signal to noise ratio enough for intended (or perhaps even unintended) learning effects to emerge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008

==== References ====
*Connelly, J., & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition and Learning, 4'', 1-29. Philadelphia, PA: Old City Publishing.
*Katz, S., Connelly, J., & Wilson, C. (2005). When should dialogues in a scaffolded learning environment take place? In P. Kommers & G. Richards (Eds.), ''Proceedings of ED-MEDIA 2005'' (pp. 2850-2855). Norfolk, VA: AACE.
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:50:36Z

Connelly: /* Explanation */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

==== Final Exam & Course Grades ====

A regression of exam score on CQPR, KCD completion counts, and problem completion counts was significant (''R''2 = .47, ''F''(3, 68) = 20.17, ''p'' < .00001) and showed positive contributions of all three factors, but only CQPR had a statistically significant effect (''p'' < .00001). However, when I omitted CQPR from the model (''R''2 = .16, ''F''(2, 72) = 7.08, ''p'' < .005), both KCD and problem completion counts reached statistical significance (''p''s < .05). A regression of final grade on all three factors was significant (''R''2 = .56, ''F''(3, 53) = 22.10, ''p'' < .00001), but again only CQPR had a statistically significant effect (p < .00001). When I omitted CQPR from this model (''R''2 = .13, ''F''(2, 55) = 3.99, ''p'' < .05), neither factor of interest reached significance, but KCD completion had a stronger marginal effect (''p'' = .073) than did problem completion (''p'' = .106).

=== Explanation ===

Despite being unable to perform many of our intended analyses, including examinations of group differences between our old Short KCDs and new Long KCDs, we were able to replicate our prior finding (Katz et al. 2007) that it was KCD completion, as opposed to the completion of target homework problems, that significantly (for most scores and subscores) improved post-test performance. In other words, whether the KCDs were Short or Long, doing more of them better predicted post-test scores than did the solving of homework problems.

Moreover, we found that both factors significantly improved scores on the final exam. That is, the more KCDs and target homework problems students did, the better they performed on the final exam. Although the effects of both factors were not statistically significant on their final course grades, the same trend existed, with KCDs being a stronger predictor of course grades that were homework problems.

This marked the first time in our line of work with KCDs (Connelly & Katz, 2006; Katz et al., 2005, 2007) that learning benefits of our dialogues transferred to longer-term performance measures. Future work could investigate the relative benefits of our Short, largely qualitative KCDs versus our Long KCDs with both qualitative and quantitative knowledge, as well as explicit ties between them.

This study also has implications for other studies of learning gains from instructional interventions, which tend to show small (if any) effects. Investigator attempts to “clean” their data, after being confronted with “cheating” and “gaming” behaviors, may be worthwhile in that they might increase the signal to noise ratio enough for intended (or perhaps even unintended) learning effects to emerge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008

==== References ====
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:48:18Z

Connelly: /* Extending Automated Dialogue Support for Robust Learning of Physics */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

==== Final Exam & Course Grades ====

A regression of exam score on CQPR, KCD completion counts, and problem completion counts was significant (''R''2 = .47, ''F''(3, 68) = 20.17, ''p'' < .00001) and showed positive contributions of all three factors, but only CQPR had a statistically significant effect (''p'' < .00001). However, when I omitted CQPR from the model (''R''2 = .16, ''F''(2, 72) = 7.08, ''p'' < .005), both KCD and problem completion counts reached statistical significance (''p''s < .05). A regression of final grade on all three factors was significant (''R''2 = .56, ''F''(3, 53) = 22.10, ''p'' < .00001), but again only CQPR had a statistically significant effect (p < .00001). When I omitted CQPR from this model (''R''2 = .13, ''F''(2, 55) = 3.99, ''p'' < .05), neither factor of interest reached significance, but KCD completion had a stronger marginal effect (''p'' = .073) than did problem completion (''p'' = .106).

=== Explanation ===

blah blah

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008

==== References ====
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:43:02Z

Connelly: /* Findings */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

==== Final Exam & Course Grades ====

A regression of exam score on CQPR, KCD completion counts, and problem completion counts was significant (''R''2 = .47, ''F''(3, 68) = 20.17, ''p'' < .00001) and showed positive contributions of all three factors, but only CQPR had a statistically significant effect (''p'' < .00001). However, when I omitted CQPR from the model (''R''2 = .16, ''F''(2, 72) = 7.08, ''p'' < .005), both KCD and problem completion counts reached statistical significance (''p''s < .05). A regression of final grade on all three factors was significant (''R''2 = .56, ''F''(3, 53) = 22.10, ''p'' < .00001), but again only CQPR had a statistically significant effect (p < .00001). When I omitted CQPR from this model (''R''2 = .13, ''F''(2, 55) = 3.99, ''p'' < .05), neither factor of interest reached significance, but KCD completion had a stronger marginal effect (''p'' = .073) than did problem completion (''p'' = .106).

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008

==== References ====
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:38:55Z

Connelly: /* Pre- and Post-Tests */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .52, ''F''(4, 54) = 14.82, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score and CQPR were statistically significant (''p''s < .0005 & .05, respectively); KCD completion was marginal (''p'' = .08) and problem completion was ns (t < 1). After omitting CQPR from the regression model (''R''2 = .50, ''F''(3, 57) = 18.74, ''p'' < .00001), the effect of KCD completion reached statistical significance (''p'' = .012) while that of problem completion remained ''ns'' (''t'' < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .37, ''F''(4, 54) = 7.96, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (''p'' < .005); KCD completion reached marginal status (''p'' = .07) only after dropping CQPR from the regression model (''R''2 = .35, ''F''(3, 57) = 10.33, ''p'' < .00001). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .46, ''F''(4, 54) = 11.53, ''p'' < .00001) showed positive contributions of all factors, but only pre-test subscore and CQPR were statistically significant (''p''s < .0005 & .01, respectively); KCD completion reached statistical significance (''p'' = .015) only after dropping CQPR from the regression model (''R''2 = .41, ''F''(3, 57) = 13.31, ''p'' < .00001). The effect of problem completion was ''ns'' in both models.

A marginal regression of normalized (Estes) gain scores on CQPR, number of KCDs completed, and number of target problems completed (''R''2 = .12, ''F''(3, 55) = 2.59, ''p'' = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (''p'' = .09). However, omitting CQPR from the model resulted in a significant regression (''R''2 = .13, ''F''(2, 58) = 4.24, ''p'' < .05) with a significant effect of KCD completion (''p'' = .016). The effect of problem completion was ''ns'' (''t'' < 1) in both models.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008

==== References ====
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:32:52Z

Connelly: /* Findings */

== Extending Automated Dialogue Support for Robust Learning of Physics ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz & John Connelly
|-
| '''Study Start Date''' || 10/1/07
|-
| '''Study End Date''' || 9/30/08
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || approx. 125 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to ''integrate'' quantitative and qualitative knowledge via [[post-practice reflection]] dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]] by trying to better support [[robust learning]] via a third condition in which students work through longer dialogues designed to foster [[transfer]] by specifically tying together qualitative and quantitative knowledge in different contexts.

Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)
* [[Transfer]]

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these [[knowledge components]] in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster [[transfer]] and [[robust learning]]?

=== Background and Significance ===
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to ''integrate'' quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, ''integrative reflective dialogues'' that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score, and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

We had also intended to measure ''short-term retention'' via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.

=== Independent Variables ===
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly, & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.

[[Image:6kcd.jpg]]

The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.

[[Image:7lkcd.jpg]]

The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

[[Image:E1b-50.jpg]]

The following student variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of viable target problems completed (before the post-test, or before the final exam)
* Number of viable (non-gibberish) dialogues completed
* Grade point average (CQPR)
* Pre-test score

=== Hypothesis ===
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these [[knowledge components]] in different problem-solving contexts would better foster [[transfer]] and [[robust learning]].

=== Findings ===
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to examine KCD and Andes logs in a much lower level of detail than for prior studies, in an attempt to determine whether each students' list of assigned problems and dialogues represented viable data (i.e., if the problems and KCDs were ''legitimately'' completed).

In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for full credit used by one course instructor).

==== Pre- and Post-Tests ====

Regressing post-test score on pre-test score, QPA, number of KCDs completed, and number of target problems completed (R2 = .52, F(4, 54) = 14.82, p < .00001) showed positive contributions of all factors, but only pre-test score and QPA were statistically significant (ps < .0005 & .05, respectively); KCD completion was marginal (p = .08) and problem completion was ns (t < 1). After omitting QPA from the regression model (R2 = .50, F(3, 57) = 18.74, p < .00001), the effect of KCD completion reached statistical significance (p = .012) while that of problem completion remained ns (t < 1).

Regressing post-test qualitative subscore on pre-test qualitative subscore, QPA, number of KCDs completed, and number of target problems completed (R2 = .37, F(4, 54) = 7.96, p < .00001) showed positive contributions of all factors, but only pre-test subscore was statistically significant (p < .005); KCD completion reached marginal status (p = .07) only after dropping QPA from the regression model (R2 = .35, F(3, 57) = 10.33, p < .00001). The effect of problem completion was ns (t < 1) in both models.

Regressing post-test quantitative subscore on pre-test quantitative subscore, QPA, number of KCDs completed, and number of target problems completed (R2 = .46, F(4, 54) = 11.53, p < .00001) showed positive contributions of all factors, but only pre-test subscore and QPA were statistically significant (ps < .0005 & .01, respectively); KCD completion reached statistical significance (p = .015) only after dropping QPA from the regression model (R2 = .41, F(3, 57) = 13.31, p < .00001). The effect of problem completion was ns in both models.

A marginal regression of normalized (Estes) gain scores on QPA, number of KCDs completed, and number of target problems completed (R2 = .12, F(3, 55) = 2.59, p = .062) showed positive contributions of all factors, but the only factor that was even marginal was KCD completion (p = .09). However, omitting QPA from the model resulted in a significant regression (R2 = .13, F(2, 58) = 4.24, p < .05) with a significant effect of KCD completion (p = .016). The effect of problem completion was ns (t < 1) in both models.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to Advisory Board, January 2008

==== References ====
*Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. ''International Journal of Artificial Intelligence and Education'', ''15''(3).
*VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), ''Artificial Intelligence in Education'' (pp. 678-685). Amsterdam, Netherlands: IOS Press.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans to wrap up the project:
* write conference paper reporting updated findings on salvaged data
* write conference paper detailing the process by which we salvaged usable data

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:25:01Z

Connelly: /* Future plans */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:23:33Z

Connelly: /* Findings */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:12:37Z

Connelly: /* Independent Variables */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:09:49Z

Connelly: /* Dependent Variables */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:06:36Z

Connelly: /* Dependent Variables */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:02:18Z

Connelly: /* Dependent Variables */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-22T22:00:44Z

Connelly: /* Abstract */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-12-03T22:28:41Z

Connelly: /* Independent Variables */

File:7lkcd.jpg

2008-12-03T22:28:06Z

Connelly:

Reflective Dialogues (Katz)

2008-11-03T20:18:13Z

Connelly: /* Independent Variables */

== Do Reflective Dialogues that Explicitly Target the “What? How? and Why (not)?” Knowledge of Physics Problem Solving Promote Expert-like Planning Ability? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz, John Connelly, Donald Treacy
|-
| '''Study Start Date''' || 3/1/06
|-
| '''Study End Date''' || 6/30/07
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 67
|-
| '''Total Participant Hours''' || approx. 750 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
One of the main differences between experts and novices in physics is that experts are more adept at identifying relevant principles and generating solution plans before starting to solve a problem (Chi, Glaser & Rees, 1982; Dufresne, Gerace, Hardiman, & Mestre, 1992; Priest & Lindsay, 1992). We propose that this difference may be due to two reasons: (1) in traditional physics courses, students are not explicitly asked to plan or are given scaffolding to support planning during problem solving, and (2) many students lack the basic knowledge of physics concepts, principles, and procedures that is prerequisite for effective planning.

In this project, we tested the effectiveness of engaging students in reflective dialogues after they solve problems in [[Andes]] that explicitly target the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why (or why not) to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). The main hypothesis tested is that explicit training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, will be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

To test this hypothesis, we conducted a Physics LearnLab study in which we compared the effectiveness of an experimental version of Andes that engaged students in reflective “what? how? and why?” dialogues after physics problem solving ([[explicit instruction]] condition) with a control version that gave students additional problem-solving practice after they solved an Andes problem—e.g., by having them identify bugs in a sample student solution to a problem ([[implicit instruction]] condition). We predicted that students in the explicit condition would outperform students in the implicit condition with respect to gain score from pre-test to post-test and scores on course exams that targeted far transfer. Consistent with prior research, the dialogues promoted conceptual understanding of physics. However, measures of retention and transfer to problem-solving ability showed only a marginal effect of completing the reflective dialogues.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge—i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them—enhance students’ problem-solving ability?

=== Background and Significance ===
Because experts develop schemata largely through a good deal of practice with solving various types of problems, traditional introductory college physics courses take a similar approach. They are structured so that students are introduced to concepts, principles, and procedures in their text and through lectures, which they are then asked to apply to many problem-solving exercises. The hope is that, with extensive practice, students will integrate [[procedural]] and [[conceptual knowledge]] and develop expert-like schemata and planning skills. Unfortunately, many students don’t. In addition to exiting these courses with lingering naïve misconceptions (e.g., Halloun & Hestenes, 1985; McDermott, 1984), they continue to solve problems largely by manipulating equations until the desired quantity is isolated—that is, by means-end analysis—instead of by identifying relevant principles and generating solution plans before starting to solve a problem, as experts do (Dufresne et al., 1992; Larkin, McDermott, Simon, & Simon, 1980; Priest & Lindsay, 1992).

We refer to the traditional approach to physics instruction described in the preceding paragraph as [[implicit instruction]], because it encourages the inductive development of abstractions (concepts, principles, and schemata) through repeated exposure to instances, instead of by explicitly reifying these abstractions (O'Malley & Chamot, 1994). In response to the limitations of implicit approaches to physics instruction that neither ask students to plan nor scaffold them in doing it, several instructional scientists have proposed methods that explicitly engage students in planning exercises (Dufresne et al., 1992; Leonard, Dufresne, & Mestre, 1996; Mestre, Dufresne, Gerace, & Hardiman, 1993). These methods have met with modest success when tested mainly using high-achieving students (B or above in an introductory college physics course). We suggest that the main reason that many students, even high achievers, are unable to use explicit planning methods effectively is that they lack the basic concepts, principles, and procedures that are prerequisites for effective planning.

This project tested the effectiveness of engaging students in reflective dialogues after they solve problems in Andes (e.g., Gertner & VanLehn, 2000) which explicitly targeted the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). Explicit instruction on these knowledge components proved to be effective and more efficient than giving students repeated practice in solving different types of problems.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''Short-term retention''. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students within each course section were block-randomly assigned to one of two dialogue conditions: standard [[Knowledge Construction Dialogues]] (KCDs) or control with no KCDs. Control students worked on standard Andes problems only:
[[Image:E1b-50.jpg]]

KCD students worked through KCDs following selected Andes problems:
[[Image:6kcd.jpg]]

Students in the control condition were assigned an additional five Andes problems to attempt to better equate time on task.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of problems completed before the post-test was administered
* Number of dialogues that the student completed
* Grade point average (CQPR)
* College major
* Pre-test score

=== Hypothesis ===
The main hypothesis tested was that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

=== Findings ===
Analyses of pre- and post-test scores were more encouraging than for last year's study ([[Post-practice reflection (Katz)]]). After omitting scores from one student with a post-test duration of less than two minutes, we were left with treatment and control groups of equal size (''n'' = 33). We also re-classified two treatment subjects who did no dialogues as control subjects (effective treatment and control ''n''s = 31 and 35, respectively), although these subjects were not able to access the five extra control-group Andes problems. As we had hoped, ANOVA showed no significant differences between the effective treatment and control groups on pre-test score (respective ''M''s = 12.10 and 11.77; ''F'' < 1), but treatment subjects had higher mean post-test scores (17.97 vs. 15.57; ''F''(1, 64) = 4.89, ''p'' = .031), mean raw gain scores (5.87 vs. 3.80; ''F'' = 5.62, ''p'' = .021), and mean Estes gain scores (0.330 vs. 0.208; ''F'' = 6.74, ''p'' = .012) than did control subjects. In short, without regard to problem-solving practice, subjects who did KCDs did significantly better on the post-test than those who did not.

Student participation in both homework problem solving and dialogues was much improved over last year, with 21 of 34 control subjects (62%) and 23 of 33 treatment subjects (70%) finishing at least 80% of the assigned target problems (of 31 for control, 26 for treatment) prior to the post-test and with 25 treatment subjects (76%) finishing at least 80% of the 26 associated KCDs. However, participation was
still far from perfect; 7 treatment subjects (21%) completed half or fewer of the assigned KCDs on time, and 2 of them completed no target problems or KCDs on time. We again treated KCD completion and target problem completion as continuous independent variables in regression analyses. Regressing post-test score on pre-test score, QPA, number of KCDs completed, and number of target problems completed
(''R''2 = .52, ''F''(4, 61) = 16.70, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score, QPA, and KCD completion were statistically significant (''p''s < .001, .05, & .05, respectively); problem completion was ''ns'' (''p'' = .54). Therefore, across all subjects it was KCD completion, as opposed to target homework problem completion, that significantly predicted post-test performance.

To measure retention and transfer, we also analyzed scores from an hourly exam administered by one instructor to his two sections (''n''=47). All problems on this exam were quantitative and covered a subset of the units targeted during the intervention. Scores on this exam ranged from 176 to 382 (of 400), and were highly correlated with
pre- and post-test scores; ''r''s(45) = .54 and .71, ''p''s < .0001 and .00001. However,
ANOVAs showed no differences between groups (Fs < 1), and regression of subscores on QPA, KCDs completed, and target problems completed (''R''2 = .54, ''F''(3, 43) = 16.58,
''p'' < .00001) showed positive contributions of all factors but a significant effect of only QPA (''p'' < .00001). Therefore, student performance on the hourly exam was not significantly affected by either KCD completion or target problem completion.

In summary, student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed prior to the post-test. However, neither measure had a significant effect on student performance on an exam that focused on
problem-solving ability.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. More students took the learning-by doing path than in our prior study, perhaps because this year instructors ''required'' them to do so (i.e., this time there were negative consequences for avoiding them).

Analyses of retention and transfer to problem-solving ability did not reveal a significant effect of the reflective dialogues. We had hoped that the capstone dialogues that treatment students completed before selected problems, at the end of each unit, would support problem-solving ability. It is possible that the intervention consisted of too few capstone dialogues (only five) to observe any effect.

What seems to be the critical feature of post-practice reflection (PPR) in supporting conceptual understanding is the explicit instruction that it provides in domain knowledge. PPR may help students to fill in knowledge gaps, resolve misconceptions,
and abstract from the case at hand so that they are better prepared to engage in constructive activity (e.g., self-explanation) in future problems. Preliminary research comparing Andes (which encourages implicit learning of problem-solving strategies) with Pyrenees, a system that teaches problem-solving strategies explicitly, also
suggests an advantage for explicit instruction (VanLehn et al., 2004). Further research is needed to identify the mechanisms that drive learning from reflective dialogue, and to increase its potential to enhance problem-solving ability in addition to conceptual knowledge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Full paper accepted at AIED 2007:
**Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.

==== References ====
*Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), ''Advances in the Psychology of Human Intelligence, Vol. 1'' (pp. 7-75). Hillsdale, NJ: Erlbaum.
*Dufresne, R. J., Gerace, P., Hardiman, T., & Mestre, J. P. (1992). Constraining novices to perform expertlike analyses: Effects on schema acquisition. ''The Journal of the Learning Sciences, 2'' (3), 307-331.
*Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), ''ITS 2000: Proceedings of the 5th International Conference on Intelligent Tutoring Systems'' (pp. 133-142). Berlin: Springer-Verlag.
*Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. ''Science, 208'', 1335-1342.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*Mestre, J. P., Dufresne, R. J., Gerace, W. J., & Hardiman, P. T. (1993). Promoting skilled problem-solving behavior among beginning physics students. ''Journal of Research in Science Teaching, 30'', 303-317.
*O’Malley, M., and Chamot, A. (1994). ''The CALLA Handbook''. Reading, MA: Addison-Wesley.
*Priest, A. G., & Lindsay, R. O. (1992). New light on novice-expert differences in physics problem solving. ''British Journal of Psychology, 83'', 389-405.
*VanLehn, K., Bhembe, D., Chi, M., Lynch, C., Schulze, K., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2004). Implicit versus explicit learning of strategies in a nonprocedural cognitive skill. In J. C. Lester, R. M. Vicari, & F. Paraguacu, (Eds.), ''Intelligent Tutoring Systems: 7th International Conference'' (pp. 521-530). Berlin: Springer-Verlag.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans for January 2008 - December 2008:
* write journal article expanding AIED conference paper

[[Category:Study]]

File:6kcd.jpg

2008-11-03T20:17:05Z

Connelly:

Post-practice reflection (Katz)

2008-11-03T20:16:42Z

Connelly: /* Independent Variables */

== Post-practice reflection in a first-year physics course: Does mixed-initiative interaction support robust learning better than tutor-led interaction or canned text? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PI''' || Sandra Katz
|-
| '''Other Contributors''' 
* '''Post-Docs:'''
| John Connelly
|-
| '''Study Start Date''' || 1/1/05
|-
| '''Study End Date''' || 12/31/05
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 123
|-
| '''Total Participant Hours''' || approx. 500 hrs
|-
| '''DataShop''' || No; Andes data still incompatible
|}

=== Abstract ===
We conducted an [[in vivo experiment]] within the PSLC Physics LearnLab to address two questions about reflection on recent problem-solving activity within [[Andes]]: (1) Does [[post-practice reflection]] support robust learning of physics—that is, students’ ability to [[transfer]] what they learned during instruction to novel situations? and (2) What is the preferred delivery mode for post-practice reflection in intelligent tutoring systems? We compared pre-test to post-test learning gains of students who were randomly assigned to one of four conditions that represent points on a scale of increasingly generative student activity: (1) a control group who solved a set of problems in Andes without any reflective activity, (2) a self-explanation group, who solved Andes problems, responded to several “[[reflection questions]]” (Lee & Hutchison, 1998) after each problem and received a canned, expert-generated explanation that they were prompted to compare with their response, (3) a “tutor-led interaction” group, who solved the same Andes problems and reflection questions (RQs) as the didactic group, but the questions were embedded in [[Knowledge Construction Dialogues]] (KCD’s) that guided students in generating a correct response, and (4) a “scaffolded mixed initiative KCD group,” which solved the same problems and responded to the same reflection questions as the other reflection groups, embedded in similar KCD’s as those presented to the “tutor-led interaction” group, with the added support of follow-up questioning menus that allowed students to take initiative.

Unfortunately, participation in the three experimental conditions was too low to compare the effectiveness of these treatments. However, a yoked pairs analysis revealed that the more reflection questions that students did, of any type, the better they did, with respect to both post-test scores and pre-test to post-test gain scores. Similarly, regression analysis revealed that the number of dialogues a student completed had a significant positive effect on post-test score, independent of the number of problems he or she completed. These findings provide empirical support for post-practice reflection in an ITS that is a central component of a course.

=== Glossary ===

* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research questions ===

Do reflection questions after physics problem solving support robust learning? Is more interactive reflection, between the automated tutor and student, more effective in supporting robust learning than less interactive reflection?

=== Background and Significance ===

Despite the well-established role that post-practice, reflective dialogue plays in apprentice-style learning (Collins, Brown, & Newman, 1989), few studies have been conducted to test their effectiveness. Hence, there is little empirical support for incorporating post-practice reflection in courses led by a human teacher or within intelligent tutoring systems (ITSs). Nor is there much guidance on how to implement reflection effectively, especially within the constraints of natural-language understanding and generation capabilities.

Research by Lee and Hutchison (1998) focused on the effectiveness of reflection questions posed after students studied worked examples on balancing chemistry equations in a computer-based learning environment. This study found that reflection questions enhanced students’ problem-solving ability. However, the role of reflection in promoting conceptual understanding was not addressed; nor did the study investigate whether reflection questions after problem-solving exercises (as opposed to example studying) support learning.

Two laboratory studies conducted by Katz and her colleagues addressed these questions. In a longitudinal analysis of student performance on avionics tasks in Sherlock 2 (e.g., Katz et al., 1998), Katz, O’Donnell and Kay (2000) found that discussions that took place between an avionics expert and two students were more effective in resolving misconceptions when they were distributed across problem solving and debrief than were discussions that took place during problem solving alone. However, due to constraints inherent in the research setting, there was no control group in this study—that is, avionics trainees who did not experience debrief led by a domain expert—and no instrument to measure performance gains. A follow-up study by Katz, Allbritton, & Connelly (2003) addressed these limitations in a different domain, first-year college physics. Forty-six students taking college physics solved problems in Andes in one of three conditions: with no reflection questions after problem solving (control group), with reflection questions discussed with human tutors, or with the same reflection questions followed by canned feedback (without a human tutor). A comparison of pre-test and post-test scores was conducted to measure learning gains. The main result was that students learned more with reflection questions than without, but the two conditions with reflection questions (canned feedback and human tutoring) did not differ significantly.

The current study is significant because it validated the results of these laboratory studies in an actual classroom setting. Specifically, it showed that post-practice reflection supports learning, as measured by pre-test to post-test learning gain scores. Due to low participation, however, this experiment did not shed light on the question of which modality of reflection is most effective.

=== Dependent Variables ===

* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near transfer ([[normal post-test]]) and far [[transfer]] (robust learning) items.

* ''Short-term retention''. Performance on course exam that covered target topic (work and energy), 1-2 weeks after the intervention.

* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students were block-randomly assigned to one of four dialogue conditions, with reflective KCDs presented after selected Andes problems:
* control (Ctrl), with no reflection questions and KCDs; just the standard Andes problems:
[[Image:E1b-50.jpg]]

* canned text response (CTR), with free-response KCDs followed by pre-scripted expert responses; students' responses were not given any positive or negative feedback:
[[Image:5ctr.jpg]]

* standard KCDs (KCD), with shorter-answer questions and tutor responses that indicated what parts of the student's response were incomplete or incorrect and tried to elicit the correct aspects:
[[Image:5kcd.jpg]]

* limited mixed-initiative KCDs (MIX), same as the standard KCDs but with on-demand hypertext question-and-answer links after selected tutor questions:
[[Image:5mix.jpg]]

However, there were no significant differences between categorical condition assignments, even after reclassifying as Ctrl subjects those in the three treatment conditions who saw no KCDs and after collapsing the KCD and MIX conditions (due to minimal usage of mixed-initiative links, for the most part the two conditions were equivalent). We therefore treated KCD and problem completion as continuous variables.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

*Number of problems completed before the post-test was administered
*Number of reflection questions that the student completed
*CQPR—grade point average
*College major grouping
*Pre-test score

=== Hypotheses ===

*Because responding to reflection questions is a form of “active learning,” students in any of the three experimental conditions (self-explanation of canned text, standard KCD’s, tutor-led KCDs) should outperform students in the control (no-reflection) condition. This hypothesis was supported.
*The more interactive the reflection modality the better, so mixed-initiative > standard KCD > self-explanation of canned text. We were unable to test this hypothesis, due to low participation.

=== Findings ===

*''Gain scores summary'':
**There were no significant differences in mean gain score (or in pre-test and post-test scores) by condition (4 conditions; 3 treatment, 1 control) or by collapsed effective condition (Ctrl, CTR, KCD).
**Yoked pairs analysis, comparing students from the same major groups with identical pre-tests scores (and minimal CQPR disparity) who completed no reflection questions with students who completed five or more questions (''n'' = 19 pairs), showed that “treated” subjects tended to out-gain “untreated” subjects.
*''Regression analysis summary'':
**The number of reflection questions completed had a significant positive effect on post-test scores.
*''Exam score summary'':
**For the final exam, there was no significant impact of the number of reflection questions completed.
**For one course section’s hourly exam on work and energy, there was a significant, positive impact of the number of reflection questions completed, but only when CQPR was dropped from the model. For the other section’s hourly exam, “treated” subjects significantly outperformed “untreated” subjects, despite being outnumbered 3 to 1.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. We are unable to determine why students chose not to take the learning-by doing path. Perhaps doing the reflection questions was too difficult for some students, but we suspect that this was not the case. A more likely explanation is that students did not have the time to complete these questions, and there were no negative consequences for avoiding them. In the follow-up study that we are currently running, students are required to complete the reflection questions in order to get credit for completing a problem.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Poster presentation at the 2006 American Association for Physics Teachers (AAPT) Conference
* Full-paper accepted at AIED 2007

==== References ====
*Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. In L. B. Resnick (Ed.), ''Knowing, learning and instruction: Essays in honor of Robert Glaser'' (pp. 543-494). Hillsdale, NJ: Erlbaum.
*Lee, A. Y., & Hutchison, L. (1998). Improving learning from examples through reflection. ''Journal of Experimental Psychology: Applied, 4'' (3), 187-210.
*Katz, S., & Allbritton, D., & Connelly, J. (2003). Going beyond the problem given: How human tutors use post-solution discussions to support transfer. ''International Journal of Artificial Intelligence and Education, 13'' (1), 79-116.
*Katz, S., Lesgold, A., Hughes, E., Peters, D., Eggan, G., Gordin, M., & Greenberg, L. (1998). Sherlock II: An intelligent tutoring system built upon the LRDC tutor framework. In C. P. Bloom & R. B. Loftin (Eds.), ''Facilitating the development and use of interactive learning environments'' (pp. 227-258). Mahwah, NJ: Erlbaum.
*Katz, S., O’Donnell, G., & Kay, H. (2000). An approach to analyzing the role and structure of reflective dialogue. ''International Journal of Artificial Intelligence and Education, 11'', 320-343.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

[[Category:Study]]

Post-practice reflection (Katz)

2008-11-03T20:10:54Z

Connelly: /* Independent Variables */

== Post-practice reflection in a first-year physics course: Does mixed-initiative interaction support robust learning better than tutor-led interaction or canned text? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PI''' || Sandra Katz
|-
| '''Other Contributors''' 
* '''Post-Docs:'''
| John Connelly
|-
| '''Study Start Date''' || 1/1/05
|-
| '''Study End Date''' || 12/31/05
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 123
|-
| '''Total Participant Hours''' || approx. 500 hrs
|-
| '''DataShop''' || No; Andes data still incompatible
|}

=== Abstract ===
We conducted an [[in vivo experiment]] within the PSLC Physics LearnLab to address two questions about reflection on recent problem-solving activity within [[Andes]]: (1) Does [[post-practice reflection]] support robust learning of physics—that is, students’ ability to [[transfer]] what they learned during instruction to novel situations? and (2) What is the preferred delivery mode for post-practice reflection in intelligent tutoring systems? We compared pre-test to post-test learning gains of students who were randomly assigned to one of four conditions that represent points on a scale of increasingly generative student activity: (1) a control group who solved a set of problems in Andes without any reflective activity, (2) a self-explanation group, who solved Andes problems, responded to several “[[reflection questions]]” (Lee & Hutchison, 1998) after each problem and received a canned, expert-generated explanation that they were prompted to compare with their response, (3) a “tutor-led interaction” group, who solved the same Andes problems and reflection questions (RQs) as the didactic group, but the questions were embedded in [[Knowledge Construction Dialogues]] (KCD’s) that guided students in generating a correct response, and (4) a “scaffolded mixed initiative KCD group,” which solved the same problems and responded to the same reflection questions as the other reflection groups, embedded in similar KCD’s as those presented to the “tutor-led interaction” group, with the added support of follow-up questioning menus that allowed students to take initiative.

Unfortunately, participation in the three experimental conditions was too low to compare the effectiveness of these treatments. However, a yoked pairs analysis revealed that the more reflection questions that students did, of any type, the better they did, with respect to both post-test scores and pre-test to post-test gain scores. Similarly, regression analysis revealed that the number of dialogues a student completed had a significant positive effect on post-test score, independent of the number of problems he or she completed. These findings provide empirical support for post-practice reflection in an ITS that is a central component of a course.

=== Glossary ===

* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research questions ===

Do reflection questions after physics problem solving support robust learning? Is more interactive reflection, between the automated tutor and student, more effective in supporting robust learning than less interactive reflection?

=== Background and Significance ===

Despite the well-established role that post-practice, reflective dialogue plays in apprentice-style learning (Collins, Brown, & Newman, 1989), few studies have been conducted to test their effectiveness. Hence, there is little empirical support for incorporating post-practice reflection in courses led by a human teacher or within intelligent tutoring systems (ITSs). Nor is there much guidance on how to implement reflection effectively, especially within the constraints of natural-language understanding and generation capabilities.

Research by Lee and Hutchison (1998) focused on the effectiveness of reflection questions posed after students studied worked examples on balancing chemistry equations in a computer-based learning environment. This study found that reflection questions enhanced students’ problem-solving ability. However, the role of reflection in promoting conceptual understanding was not addressed; nor did the study investigate whether reflection questions after problem-solving exercises (as opposed to example studying) support learning.

Two laboratory studies conducted by Katz and her colleagues addressed these questions. In a longitudinal analysis of student performance on avionics tasks in Sherlock 2 (e.g., Katz et al., 1998), Katz, O’Donnell and Kay (2000) found that discussions that took place between an avionics expert and two students were more effective in resolving misconceptions when they were distributed across problem solving and debrief than were discussions that took place during problem solving alone. However, due to constraints inherent in the research setting, there was no control group in this study—that is, avionics trainees who did not experience debrief led by a domain expert—and no instrument to measure performance gains. A follow-up study by Katz, Allbritton, & Connelly (2003) addressed these limitations in a different domain, first-year college physics. Forty-six students taking college physics solved problems in Andes in one of three conditions: with no reflection questions after problem solving (control group), with reflection questions discussed with human tutors, or with the same reflection questions followed by canned feedback (without a human tutor). A comparison of pre-test and post-test scores was conducted to measure learning gains. The main result was that students learned more with reflection questions than without, but the two conditions with reflection questions (canned feedback and human tutoring) did not differ significantly.

The current study is significant because it validated the results of these laboratory studies in an actual classroom setting. Specifically, it showed that post-practice reflection supports learning, as measured by pre-test to post-test learning gain scores. Due to low participation, however, this experiment did not shed light on the question of which modality of reflection is most effective.

=== Dependent Variables ===

* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near transfer ([[normal post-test]]) and far [[transfer]] (robust learning) items.

* ''Short-term retention''. Performance on course exam that covered target topic (work and energy), 1-2 weeks after the intervention.

* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students were block-randomly assigned to one of four dialogue conditions, with reflective KCDs presented after selected Andes problems:
* control (Ctrl), with no reflection questions and KCDs; just the standard Andes problems:
[[Image:E1b-50.jpg]]
* canned text response (CTR), with free-response KCDs followed by pre-scripted expert responses; students' responses were not given any positive or negative feedback:
[[Image:5ctr.jpg]]
* standard KCDs (KCD), with shorter-answer questions and tutor responses that indicated what parts of the student's response were incomplete or incorrect and tried to elicit the correct aspects:
[[Image:5kcd.jpg]]
* limited mixed-initiative KCDs (MIX), same as the standard KCDs but with on-demand hypertext question-and-answer links after selected tutor questions:
[[Image:5mix.jpg]]

However, there were no significant differences between categorical condition assignments, even after reclassifying as Ctrl subjects those in the three treatment conditions who saw no KCDs and after collapsing the KCD and MIX conditions (due to minimal usage of mixed-initiative links, for the most part the two conditions were equivalent). We therefore treated KCD and problem completion as continuous variables.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

*Number of problems completed before the post-test was administered
*Number of reflection questions that the student completed
*CQPR—grade point average
*College major grouping
*Pre-test score

=== Hypotheses ===

*Because responding to reflection questions is a form of “active learning,” students in any of the three experimental conditions (self-explanation of canned text, standard KCD’s, tutor-led KCDs) should outperform students in the control (no-reflection) condition. This hypothesis was supported.
*The more interactive the reflection modality the better, so mixed-initiative > standard KCD > self-explanation of canned text. We were unable to test this hypothesis, due to low participation.

=== Findings ===

*''Gain scores summary'':
**There were no significant differences in mean gain score (or in pre-test and post-test scores) by condition (4 conditions; 3 treatment, 1 control) or by collapsed effective condition (Ctrl, CTR, KCD).
**Yoked pairs analysis, comparing students from the same major groups with identical pre-tests scores (and minimal CQPR disparity) who completed no reflection questions with students who completed five or more questions (''n'' = 19 pairs), showed that “treated” subjects tended to out-gain “untreated” subjects.
*''Regression analysis summary'':
**The number of reflection questions completed had a significant positive effect on post-test scores.
*''Exam score summary'':
**For the final exam, there was no significant impact of the number of reflection questions completed.
**For one course section’s hourly exam on work and energy, there was a significant, positive impact of the number of reflection questions completed, but only when CQPR was dropped from the model. For the other section’s hourly exam, “treated” subjects significantly outperformed “untreated” subjects, despite being outnumbered 3 to 1.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. We are unable to determine why students chose not to take the learning-by doing path. Perhaps doing the reflection questions was too difficult for some students, but we suspect that this was not the case. A more likely explanation is that students did not have the time to complete these questions, and there were no negative consequences for avoiding them. In the follow-up study that we are currently running, students are required to complete the reflection questions in order to get credit for completing a problem.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Poster presentation at the 2006 American Association for Physics Teachers (AAPT) Conference
* Full-paper accepted at AIED 2007

==== References ====
*Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. In L. B. Resnick (Ed.), ''Knowing, learning and instruction: Essays in honor of Robert Glaser'' (pp. 543-494). Hillsdale, NJ: Erlbaum.
*Lee, A. Y., & Hutchison, L. (1998). Improving learning from examples through reflection. ''Journal of Experimental Psychology: Applied, 4'' (3), 187-210.
*Katz, S., & Allbritton, D., & Connelly, J. (2003). Going beyond the problem given: How human tutors use post-solution discussions to support transfer. ''International Journal of Artificial Intelligence and Education, 13'' (1), 79-116.
*Katz, S., Lesgold, A., Hughes, E., Peters, D., Eggan, G., Gordin, M., & Greenberg, L. (1998). Sherlock II: An intelligent tutoring system built upon the LRDC tutor framework. In C. P. Bloom & R. B. Loftin (Eds.), ''Facilitating the development and use of interactive learning environments'' (pp. 227-258). Mahwah, NJ: Erlbaum.
*Katz, S., O’Donnell, G., & Kay, H. (2000). An approach to analyzing the role and structure of reflective dialogue. ''International Journal of Artificial Intelligence and Education, 11'', 320-343.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

[[Category:Study]]

Post-practice reflection (Katz)

2008-11-03T20:05:34Z

Connelly: /* Independent Variables */

== Post-practice reflection in a first-year physics course: Does mixed-initiative interaction support robust learning better than tutor-led interaction or canned text? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PI''' || Sandra Katz
|-
| '''Other Contributors''' 
* '''Post-Docs:'''
| John Connelly
|-
| '''Study Start Date''' || 1/1/05
|-
| '''Study End Date''' || 12/31/05
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 123
|-
| '''Total Participant Hours''' || approx. 500 hrs
|-
| '''DataShop''' || No; Andes data still incompatible
|}

=== Abstract ===
We conducted an [[in vivo experiment]] within the PSLC Physics LearnLab to address two questions about reflection on recent problem-solving activity within [[Andes]]: (1) Does [[post-practice reflection]] support robust learning of physics—that is, students’ ability to [[transfer]] what they learned during instruction to novel situations? and (2) What is the preferred delivery mode for post-practice reflection in intelligent tutoring systems? We compared pre-test to post-test learning gains of students who were randomly assigned to one of four conditions that represent points on a scale of increasingly generative student activity: (1) a control group who solved a set of problems in Andes without any reflective activity, (2) a self-explanation group, who solved Andes problems, responded to several “[[reflection questions]]” (Lee & Hutchison, 1998) after each problem and received a canned, expert-generated explanation that they were prompted to compare with their response, (3) a “tutor-led interaction” group, who solved the same Andes problems and reflection questions (RQs) as the didactic group, but the questions were embedded in [[Knowledge Construction Dialogues]] (KCD’s) that guided students in generating a correct response, and (4) a “scaffolded mixed initiative KCD group,” which solved the same problems and responded to the same reflection questions as the other reflection groups, embedded in similar KCD’s as those presented to the “tutor-led interaction” group, with the added support of follow-up questioning menus that allowed students to take initiative.

Unfortunately, participation in the three experimental conditions was too low to compare the effectiveness of these treatments. However, a yoked pairs analysis revealed that the more reflection questions that students did, of any type, the better they did, with respect to both post-test scores and pre-test to post-test gain scores. Similarly, regression analysis revealed that the number of dialogues a student completed had a significant positive effect on post-test score, independent of the number of problems he or she completed. These findings provide empirical support for post-practice reflection in an ITS that is a central component of a course.

=== Glossary ===

* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research questions ===

Do reflection questions after physics problem solving support robust learning? Is more interactive reflection, between the automated tutor and student, more effective in supporting robust learning than less interactive reflection?

=== Background and Significance ===

Despite the well-established role that post-practice, reflective dialogue plays in apprentice-style learning (Collins, Brown, & Newman, 1989), few studies have been conducted to test their effectiveness. Hence, there is little empirical support for incorporating post-practice reflection in courses led by a human teacher or within intelligent tutoring systems (ITSs). Nor is there much guidance on how to implement reflection effectively, especially within the constraints of natural-language understanding and generation capabilities.

Research by Lee and Hutchison (1998) focused on the effectiveness of reflection questions posed after students studied worked examples on balancing chemistry equations in a computer-based learning environment. This study found that reflection questions enhanced students’ problem-solving ability. However, the role of reflection in promoting conceptual understanding was not addressed; nor did the study investigate whether reflection questions after problem-solving exercises (as opposed to example studying) support learning.

Two laboratory studies conducted by Katz and her colleagues addressed these questions. In a longitudinal analysis of student performance on avionics tasks in Sherlock 2 (e.g., Katz et al., 1998), Katz, O’Donnell and Kay (2000) found that discussions that took place between an avionics expert and two students were more effective in resolving misconceptions when they were distributed across problem solving and debrief than were discussions that took place during problem solving alone. However, due to constraints inherent in the research setting, there was no control group in this study—that is, avionics trainees who did not experience debrief led by a domain expert—and no instrument to measure performance gains. A follow-up study by Katz, Allbritton, & Connelly (2003) addressed these limitations in a different domain, first-year college physics. Forty-six students taking college physics solved problems in Andes in one of three conditions: with no reflection questions after problem solving (control group), with reflection questions discussed with human tutors, or with the same reflection questions followed by canned feedback (without a human tutor). A comparison of pre-test and post-test scores was conducted to measure learning gains. The main result was that students learned more with reflection questions than without, but the two conditions with reflection questions (canned feedback and human tutoring) did not differ significantly.

The current study is significant because it validated the results of these laboratory studies in an actual classroom setting. Specifically, it showed that post-practice reflection supports learning, as measured by pre-test to post-test learning gain scores. Due to low participation, however, this experiment did not shed light on the question of which modality of reflection is most effective.

=== Dependent Variables ===

* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near transfer ([[normal post-test]]) and far [[transfer]] (robust learning) items.

* ''Short-term retention''. Performance on course exam that covered target topic (work and energy), 1-2 weeks after the intervention.

* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students were block-randomly assigned to one of four dialogue conditions:
* control (Ctrl), with no reflection questions and KCDs; just standard Andes problems:
[[Image:E1b-50.jpg]]
* canned text response (CTR), with free-response KCDs (after each target Andes problem) followed by pre-scripted expert responses; students' responses were not given any positive/negative feedback:
[[Image:5ctr.jpg]]
* standard KCDs (KCD), with shorter-answer questions and tutor responses that indicate what parts of the student's response was incomplete or incorrect and try to elicit the correct aspects:
[[Image:5kcd.jpg]]
* limited mixed-initiative KCDs (MIX), same as the standard KCDs but with on-demand hypertext question-and-answer links after selected tutor questions:
[[Image:5mix.jpg]]

However, there were no significant differences between categorical condition assignments, even after reclassifying as Ctrl subjects those in the three treatment conditions who saw no KCDs and after collapsing the KCD and MIX conditions (due to minimal usage of mixed-initiative links, for the most part the two conditions were equivalent). We therefore treated KCD and problem completion as continuous variables.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

*Number of problems completed before the post-test was administered
*Number of reflection questions that the student completed
*CQPR—grade point average
*College major grouping
*Pre-test score

=== Hypotheses ===

*Because responding to reflection questions is a form of “active learning,” students in any of the three experimental conditions (self-explanation of canned text, standard KCD’s, tutor-led KCDs) should outperform students in the control (no-reflection) condition. This hypothesis was supported.
*The more interactive the reflection modality the better, so mixed-initiative > standard KCD > self-explanation of canned text. We were unable to test this hypothesis, due to low participation.

=== Findings ===

*''Gain scores summary'':
**There were no significant differences in mean gain score (or in pre-test and post-test scores) by condition (4 conditions; 3 treatment, 1 control) or by collapsed effective condition (Ctrl, CTR, KCD).
**Yoked pairs analysis, comparing students from the same major groups with identical pre-tests scores (and minimal CQPR disparity) who completed no reflection questions with students who completed five or more questions (''n'' = 19 pairs), showed that “treated” subjects tended to out-gain “untreated” subjects.
*''Regression analysis summary'':
**The number of reflection questions completed had a significant positive effect on post-test scores.
*''Exam score summary'':
**For the final exam, there was no significant impact of the number of reflection questions completed.
**For one course section’s hourly exam on work and energy, there was a significant, positive impact of the number of reflection questions completed, but only when CQPR was dropped from the model. For the other section’s hourly exam, “treated” subjects significantly outperformed “untreated” subjects, despite being outnumbered 3 to 1.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. We are unable to determine why students chose not to take the learning-by doing path. Perhaps doing the reflection questions was too difficult for some students, but we suspect that this was not the case. A more likely explanation is that students did not have the time to complete these questions, and there were no negative consequences for avoiding them. In the follow-up study that we are currently running, students are required to complete the reflection questions in order to get credit for completing a problem.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Poster presentation at the 2006 American Association for Physics Teachers (AAPT) Conference
* Full-paper accepted at AIED 2007

==== References ====
*Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. In L. B. Resnick (Ed.), ''Knowing, learning and instruction: Essays in honor of Robert Glaser'' (pp. 543-494). Hillsdale, NJ: Erlbaum.
*Lee, A. Y., & Hutchison, L. (1998). Improving learning from examples through reflection. ''Journal of Experimental Psychology: Applied, 4'' (3), 187-210.
*Katz, S., & Allbritton, D., & Connelly, J. (2003). Going beyond the problem given: How human tutors use post-solution discussions to support transfer. ''International Journal of Artificial Intelligence and Education, 13'' (1), 79-116.
*Katz, S., Lesgold, A., Hughes, E., Peters, D., Eggan, G., Gordin, M., & Greenberg, L. (1998). Sherlock II: An intelligent tutoring system built upon the LRDC tutor framework. In C. P. Bloom & R. B. Loftin (Eds.), ''Facilitating the development and use of interactive learning environments'' (pp. 227-258). Mahwah, NJ: Erlbaum.
*Katz, S., O’Donnell, G., & Kay, H. (2000). An approach to analyzing the role and structure of reflective dialogue. ''International Journal of Artificial Intelligence and Education, 11'', 320-343.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

[[Category:Study]]

File:5mix.jpg

2008-11-03T20:01:12Z

Connelly:

File:5kcd.jpg

2008-11-03T20:00:33Z

Connelly:

File:5ctr.jpg

2008-11-03T19:59:42Z

Connelly:

Post-practice reflection (Katz)

2008-10-09T19:51:10Z

Connelly: /* Independent Variables */

== Post-practice reflection in a first-year physics course: Does mixed-initiative interaction support robust learning better than tutor-led interaction or canned text? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PI''' || Sandra Katz
|-
| '''Other Contributors''' 
* '''Post-Docs:'''
| John Connelly
|-
| '''Study Start Date''' || 1/1/05
|-
| '''Study End Date''' || 12/31/05
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 123
|-
| '''Total Participant Hours''' || approx. 500 hrs
|-
| '''DataShop''' || No; Andes data still incompatible
|}

=== Abstract ===
We conducted an [[in vivo experiment]] within the PSLC Physics LearnLab to address two questions about reflection on recent problem-solving activity within [[Andes]]: (1) Does [[post-practice reflection]] support robust learning of physics—that is, students’ ability to [[transfer]] what they learned during instruction to novel situations? and (2) What is the preferred delivery mode for post-practice reflection in intelligent tutoring systems? We compared pre-test to post-test learning gains of students who were randomly assigned to one of four conditions that represent points on a scale of increasingly generative student activity: (1) a control group who solved a set of problems in Andes without any reflective activity, (2) a self-explanation group, who solved Andes problems, responded to several “[[reflection questions]]” (Lee & Hutchison, 1998) after each problem and received a canned, expert-generated explanation that they were prompted to compare with their response, (3) a “tutor-led interaction” group, who solved the same Andes problems and reflection questions (RQs) as the didactic group, but the questions were embedded in [[Knowledge Construction Dialogues]] (KCD’s) that guided students in generating a correct response, and (4) a “scaffolded mixed initiative KCD group,” which solved the same problems and responded to the same reflection questions as the other reflection groups, embedded in similar KCD’s as those presented to the “tutor-led interaction” group, with the added support of follow-up questioning menus that allowed students to take initiative.

Unfortunately, participation in the three experimental conditions was too low to compare the effectiveness of these treatments. However, a yoked pairs analysis revealed that the more reflection questions that students did, of any type, the better they did, with respect to both post-test scores and pre-test to post-test gain scores. Similarly, regression analysis revealed that the number of dialogues a student completed had a significant positive effect on post-test score, independent of the number of problems he or she completed. These findings provide empirical support for post-practice reflection in an ITS that is a central component of a course.

=== Glossary ===

* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research questions ===

Do reflection questions after physics problem solving support robust learning? Is more interactive reflection, between the automated tutor and student, more effective in supporting robust learning than less interactive reflection?

=== Background and Significance ===

Despite the well-established role that post-practice, reflective dialogue plays in apprentice-style learning (Collins, Brown, & Newman, 1989), few studies have been conducted to test their effectiveness. Hence, there is little empirical support for incorporating post-practice reflection in courses led by a human teacher or within intelligent tutoring systems (ITSs). Nor is there much guidance on how to implement reflection effectively, especially within the constraints of natural-language understanding and generation capabilities.

Research by Lee and Hutchison (1998) focused on the effectiveness of reflection questions posed after students studied worked examples on balancing chemistry equations in a computer-based learning environment. This study found that reflection questions enhanced students’ problem-solving ability. However, the role of reflection in promoting conceptual understanding was not addressed; nor did the study investigate whether reflection questions after problem-solving exercises (as opposed to example studying) support learning.

Two laboratory studies conducted by Katz and her colleagues addressed these questions. In a longitudinal analysis of student performance on avionics tasks in Sherlock 2 (e.g., Katz et al., 1998), Katz, O’Donnell and Kay (2000) found that discussions that took place between an avionics expert and two students were more effective in resolving misconceptions when they were distributed across problem solving and debrief than were discussions that took place during problem solving alone. However, due to constraints inherent in the research setting, there was no control group in this study—that is, avionics trainees who did not experience debrief led by a domain expert—and no instrument to measure performance gains. A follow-up study by Katz, Allbritton, & Connelly (2003) addressed these limitations in a different domain, first-year college physics. Forty-six students taking college physics solved problems in Andes in one of three conditions: with no reflection questions after problem solving (control group), with reflection questions discussed with human tutors, or with the same reflection questions followed by canned feedback (without a human tutor). A comparison of pre-test and post-test scores was conducted to measure learning gains. The main result was that students learned more with reflection questions than without, but the two conditions with reflection questions (canned feedback and human tutoring) did not differ significantly.

The current study is significant because it validated the results of these laboratory studies in an actual classroom setting. Specifically, it showed that post-practice reflection supports learning, as measured by pre-test to post-test learning gain scores. Due to low participation, however, this experiment did not shed light on the question of which modality of reflection is most effective.

=== Dependent Variables ===

* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near transfer ([[normal post-test]]) and far [[transfer]] (robust learning) items.

* ''Short-term retention''. Performance on course exam that covered target topic (work and energy), 1-2 weeks after the intervention.

* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students were block-randomly assigned to one of four dialogue conditions:
* control (Ctrl), with no reflection questions and KCDs;
[[Image:E1b-50.jpg]]
* canned text response (CTR), with free-response KCDs followed by pre-scripted expert responses; students' responses were not given any positive/negative feedback.
* standard KCDs (KCD), with shorter-answer questions and tutor responses that indicate what parts of the student's response was incomplete or incorrect and try to elicit the correct aspects; and
* limited mixed-initiative (MIX), same as the standard KCDs, but with on-demand hypertext question-and-answer links.

However, there were no significant differences between categorical condition assignments, even after reclassifying as Ctrl subjects those in the three treatment conditions who saw no KCDs and after collapsing the KCD and MIX conditions (due to minimal usage of mixed-initiative links, for the most part the two conditions were equivalent). We therefore treated KCD and problem completion as continuous variables.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

*Number of problems completed before the post-test was administered
*Number of reflection questions that the student completed
*CQPR—grade point average
*College major grouping
*Pre-test score

=== Hypotheses ===

*Because responding to reflection questions is a form of “active learning,” students in any of the three experimental conditions (self-explanation of canned text, standard KCD’s, tutor-led KCDs) should outperform students in the control (no-reflection) condition. This hypothesis was supported.
*The more interactive the reflection modality the better, so mixed-initiative > standard KCD > self-explanation of canned text. We were unable to test this hypothesis, due to low participation.

=== Findings ===

*''Gain scores summary'':
**There were no significant differences in mean gain score (or in pre-test and post-test scores) by condition (4 conditions; 3 treatment, 1 control) or by collapsed effective condition (Ctrl, CTR, KCD).
**Yoked pairs analysis, comparing students from the same major groups with identical pre-tests scores (and minimal CQPR disparity) who completed no reflection questions with students who completed five or more questions (''n'' = 19 pairs), showed that “treated” subjects tended to out-gain “untreated” subjects.
*''Regression analysis summary'':
**The number of reflection questions completed had a significant positive effect on post-test scores.
*''Exam score summary'':
**For the final exam, there was no significant impact of the number of reflection questions completed.
**For one course section’s hourly exam on work and energy, there was a significant, positive impact of the number of reflection questions completed, but only when CQPR was dropped from the model. For the other section’s hourly exam, “treated” subjects significantly outperformed “untreated” subjects, despite being outnumbered 3 to 1.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. We are unable to determine why students chose not to take the learning-by doing path. Perhaps doing the reflection questions was too difficult for some students, but we suspect that this was not the case. A more likely explanation is that students did not have the time to complete these questions, and there were no negative consequences for avoiding them. In the follow-up study that we are currently running, students are required to complete the reflection questions in order to get credit for completing a problem.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Poster presentation at the 2006 American Association for Physics Teachers (AAPT) Conference
* Full-paper accepted at AIED 2007

==== References ====
*Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. In L. B. Resnick (Ed.), ''Knowing, learning and instruction: Essays in honor of Robert Glaser'' (pp. 543-494). Hillsdale, NJ: Erlbaum.
*Lee, A. Y., & Hutchison, L. (1998). Improving learning from examples through reflection. ''Journal of Experimental Psychology: Applied, 4'' (3), 187-210.
*Katz, S., & Allbritton, D., & Connelly, J. (2003). Going beyond the problem given: How human tutors use post-solution discussions to support transfer. ''International Journal of Artificial Intelligence and Education, 13'' (1), 79-116.
*Katz, S., Lesgold, A., Hughes, E., Peters, D., Eggan, G., Gordin, M., & Greenberg, L. (1998). Sherlock II: An intelligent tutoring system built upon the LRDC tutor framework. In C. P. Bloom & R. B. Loftin (Eds.), ''Facilitating the development and use of interactive learning environments'' (pp. 227-258). Mahwah, NJ: Erlbaum.
*Katz, S., O’Donnell, G., & Kay, H. (2000). An approach to analyzing the role and structure of reflective dialogue. ''International Journal of Artificial Intelligence and Education, 11'', 320-343.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

[[Category:Study]]

File:E1b-25.jpg

2008-10-09T19:49:54Z

Connelly:

File:E1b-50.jpg

2008-10-09T19:47:53Z

Connelly:

File:E1b.JPG

2008-10-09T19:45:07Z

Connelly:

Post-practice reflection (Katz)

2008-10-09T19:28:27Z

Connelly: /* Independent Variables */

== Post-practice reflection in a first-year physics course: Does mixed-initiative interaction support robust learning better than tutor-led interaction or canned text? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PI''' || Sandra Katz
|-
| '''Other Contributors''' 
* '''Post-Docs:'''
| John Connelly
|-
| '''Study Start Date''' || 1/1/05
|-
| '''Study End Date''' || 12/31/05
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 123
|-
| '''Total Participant Hours''' || approx. 500 hrs
|-
| '''DataShop''' || No; Andes data still incompatible
|}

=== Abstract ===
We conducted an [[in vivo experiment]] within the PSLC Physics LearnLab to address two questions about reflection on recent problem-solving activity within [[Andes]]: (1) Does [[post-practice reflection]] support robust learning of physics—that is, students’ ability to [[transfer]] what they learned during instruction to novel situations? and (2) What is the preferred delivery mode for post-practice reflection in intelligent tutoring systems? We compared pre-test to post-test learning gains of students who were randomly assigned to one of four conditions that represent points on a scale of increasingly generative student activity: (1) a control group who solved a set of problems in Andes without any reflective activity, (2) a self-explanation group, who solved Andes problems, responded to several “[[reflection questions]]” (Lee & Hutchison, 1998) after each problem and received a canned, expert-generated explanation that they were prompted to compare with their response, (3) a “tutor-led interaction” group, who solved the same Andes problems and reflection questions (RQs) as the didactic group, but the questions were embedded in [[Knowledge Construction Dialogues]] (KCD’s) that guided students in generating a correct response, and (4) a “scaffolded mixed initiative KCD group,” which solved the same problems and responded to the same reflection questions as the other reflection groups, embedded in similar KCD’s as those presented to the “tutor-led interaction” group, with the added support of follow-up questioning menus that allowed students to take initiative.

Unfortunately, participation in the three experimental conditions was too low to compare the effectiveness of these treatments. However, a yoked pairs analysis revealed that the more reflection questions that students did, of any type, the better they did, with respect to both post-test scores and pre-test to post-test gain scores. Similarly, regression analysis revealed that the number of dialogues a student completed had a significant positive effect on post-test score, independent of the number of problems he or she completed. These findings provide empirical support for post-practice reflection in an ITS that is a central component of a course.

=== Glossary ===

* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research questions ===

Do reflection questions after physics problem solving support robust learning? Is more interactive reflection, between the automated tutor and student, more effective in supporting robust learning than less interactive reflection?

=== Background and Significance ===

Despite the well-established role that post-practice, reflective dialogue plays in apprentice-style learning (Collins, Brown, & Newman, 1989), few studies have been conducted to test their effectiveness. Hence, there is little empirical support for incorporating post-practice reflection in courses led by a human teacher or within intelligent tutoring systems (ITSs). Nor is there much guidance on how to implement reflection effectively, especially within the constraints of natural-language understanding and generation capabilities.

Research by Lee and Hutchison (1998) focused on the effectiveness of reflection questions posed after students studied worked examples on balancing chemistry equations in a computer-based learning environment. This study found that reflection questions enhanced students’ problem-solving ability. However, the role of reflection in promoting conceptual understanding was not addressed; nor did the study investigate whether reflection questions after problem-solving exercises (as opposed to example studying) support learning.

Two laboratory studies conducted by Katz and her colleagues addressed these questions. In a longitudinal analysis of student performance on avionics tasks in Sherlock 2 (e.g., Katz et al., 1998), Katz, O’Donnell and Kay (2000) found that discussions that took place between an avionics expert and two students were more effective in resolving misconceptions when they were distributed across problem solving and debrief than were discussions that took place during problem solving alone. However, due to constraints inherent in the research setting, there was no control group in this study—that is, avionics trainees who did not experience debrief led by a domain expert—and no instrument to measure performance gains. A follow-up study by Katz, Allbritton, & Connelly (2003) addressed these limitations in a different domain, first-year college physics. Forty-six students taking college physics solved problems in Andes in one of three conditions: with no reflection questions after problem solving (control group), with reflection questions discussed with human tutors, or with the same reflection questions followed by canned feedback (without a human tutor). A comparison of pre-test and post-test scores was conducted to measure learning gains. The main result was that students learned more with reflection questions than without, but the two conditions with reflection questions (canned feedback and human tutoring) did not differ significantly.

The current study is significant because it validated the results of these laboratory studies in an actual classroom setting. Specifically, it showed that post-practice reflection supports learning, as measured by pre-test to post-test learning gain scores. Due to low participation, however, this experiment did not shed light on the question of which modality of reflection is most effective.

=== Dependent Variables ===

* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near transfer ([[normal post-test]]) and far [[transfer]] (robust learning) items.

* ''Short-term retention''. Performance on course exam that covered target topic (work and energy), 1-2 weeks after the intervention.

* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students were block-randomly assigned to one of four dialogue conditions:
* control (Ctrl), with no reflection questions and KCDs;
* canned text response (CTR), with free-response KCDs followed by pre-scripted expert responses; students' responses were not given any positive/negative feedback.
* standard KCDs (KCD), with shorter-answer questions and tutor responses that indicate what parts of the student's response was incomplete or incorrect and try to elicit the correct aspects; and
* limited mixed-initiative (MIX), same as the standard KCDs, but with on-demand hypertext question-and-answer links.

However, there were no significant differences between categorical condition assignments, even after reclassifying as Ctrl subjects those in the three treatment conditions who saw no KCDs and after collapsing the KCD and MIX conditions (due to minimal usage of mixed-initiative links, for the most part the two conditions were equivalent). We therefore treated KCD and problem completion as continuous variables.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

*Number of problems completed before the post-test was administered
*Number of reflection questions that the student completed
*CQPR—grade point average
*College major grouping
*Pre-test score

=== Hypotheses ===

*Because responding to reflection questions is a form of “active learning,” students in any of the three experimental conditions (self-explanation of canned text, standard KCD’s, tutor-led KCDs) should outperform students in the control (no-reflection) condition. This hypothesis was supported.
*The more interactive the reflection modality the better, so mixed-initiative > standard KCD > self-explanation of canned text. We were unable to test this hypothesis, due to low participation.

=== Findings ===

*''Gain scores summary'':
**There were no significant differences in mean gain score (or in pre-test and post-test scores) by condition (4 conditions; 3 treatment, 1 control) or by collapsed effective condition (Ctrl, CTR, KCD).
**Yoked pairs analysis, comparing students from the same major groups with identical pre-tests scores (and minimal CQPR disparity) who completed no reflection questions with students who completed five or more questions (''n'' = 19 pairs), showed that “treated” subjects tended to out-gain “untreated” subjects.
*''Regression analysis summary'':
**The number of reflection questions completed had a significant positive effect on post-test scores.
*''Exam score summary'':
**For the final exam, there was no significant impact of the number of reflection questions completed.
**For one course section’s hourly exam on work and energy, there was a significant, positive impact of the number of reflection questions completed, but only when CQPR was dropped from the model. For the other section’s hourly exam, “treated” subjects significantly outperformed “untreated” subjects, despite being outnumbered 3 to 1.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. We are unable to determine why students chose not to take the learning-by doing path. Perhaps doing the reflection questions was too difficult for some students, but we suspect that this was not the case. A more likely explanation is that students did not have the time to complete these questions, and there were no negative consequences for avoiding them. In the follow-up study that we are currently running, students are required to complete the reflection questions in order to get credit for completing a problem.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Poster presentation at the 2006 American Association for Physics Teachers (AAPT) Conference
* Full-paper accepted at AIED 2007

==== References ====
*Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. In L. B. Resnick (Ed.), ''Knowing, learning and instruction: Essays in honor of Robert Glaser'' (pp. 543-494). Hillsdale, NJ: Erlbaum.
*Lee, A. Y., & Hutchison, L. (1998). Improving learning from examples through reflection. ''Journal of Experimental Psychology: Applied, 4'' (3), 187-210.
*Katz, S., & Allbritton, D., & Connelly, J. (2003). Going beyond the problem given: How human tutors use post-solution discussions to support transfer. ''International Journal of Artificial Intelligence and Education, 13'' (1), 79-116.
*Katz, S., Lesgold, A., Hughes, E., Peters, D., Eggan, G., Gordin, M., & Greenberg, L. (1998). Sherlock II: An intelligent tutoring system built upon the LRDC tutor framework. In C. P. Bloom & R. B. Loftin (Eds.), ''Facilitating the development and use of interactive learning environments'' (pp. 227-258). Mahwah, NJ: Erlbaum.
*Katz, S., O’Donnell, G., & Kay, H. (2000). An approach to analyzing the role and structure of reflective dialogue. ''International Journal of Artificial Intelligence and Education, 11'', 320-343.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T21:33:54Z

Connelly: /* Future plans */

Interactive Communication

2008-04-15T21:29:24Z

Connelly: /* Questioning */

= The PSLC Interactive Communication cluster =

== Abstract ==
The studies in the Interactive Communication deal primarily with learning environments where there are two interacting, communicating agents, one of which is the student. The other [[agent]] is typically a second student, a human tutor or a tutoring system. They communicate, either in a natural language or a formal language, such as mathematical expression or menus. We are trying to find out why such instructional, dyadic, interactive communication is sometimes highly effective and sometimes less effective. Sometimes we study highly constrained forms of communication in order to vary isolated aspects, and sometimes we compare whole forms of communciation. Our hypothesis is simply that interactive communication is effective if it guides students to attend to the right [[knowledge components]].

 <center>[[Image:ic-theory.jpg]]</center>

== Background and Significance ==
Although instructional dialogue has been studied in classrooms (e.g., Lave & Wenger, 1991; Leinhardt, 1990) and workplaces (e.g., Hutchins, 1995; Nunes, Schliemann & Carraher, 1993), we are focusing on more tractable albeit still complex situations: ''dyadic'' instructional dialogues, namely dialogues between: (a) a human tutor and a human student, (b) two human students, or (c) A computer tutor and a human student. Moreover, the dialogue are task-oriented (Grosz & Sidner, 1986) in that the participants are working together on a task rather than simply conversing with no shared goals or with opposing goals.

Early studies focused on the structure of dyadic instructional dialogue (e.g., Fox, 1993; Graesser, Person & Magliano, 1995; MacArthur, Stasz, & Zmuidzinas, 1990). When later studies compared the learning that occurred during dialogue vs. less interactive instruction (e.g., VanLehn, Graesser et al., 2007; Katz, Connelly & Allbritton, 2003; Evens & Michael, 2006; Cohen, Kulik & Kulik, 1982), they found surprisingly mixed results. Only 60% of the studies showed that interactive communication caused larger learning gains than less interactive instruction.

The interactive communication cluster is undertaking the next step in this important line of research by investigating when different types of interactive communication are effective and why. Sometimes we compare highly constrained forms of communication in order to vary isolated aspects, and sometimes we compare constrained interactive communication to passive communication (e.g., reading).

== Glossary ==
See [[:Category:Interactive Communication|Interactive Communication Glossary]]

== Research question ==
What properties of interactive communication promote robust learning?

== Independent variables ==
The independent variables (also called Treatment Variables) of the IC cluster appear as column headers in the matrix above. They are listed here with links to their glossary entries.

* [[Collaboration]]

* [[Vicarious learning]]

* [[Collaboration scripts]]

* [[Deep/Reflection questions]]

* [[Instructional explanation]]

* [[Prompted Self-explanation]]

* [[Tutoring feedback]]

* [[Error correction support]]

== Dependent variables ==
Measures of normal and robust learning.

== Hypothesis ==
Our central hypothesis is just a special case of the [[Knowledge component hypothesis]]: interactive communication is effective if it guides students to attend to the right [[knowledge components]]. The key words here are “guide” and “attend” because they may oppose each other. A dialogue that strongly guides the student may also cause the student to disengage and thus not attend to the knowledge component even if the student’s dialogue partner mentions them. On the other hand, an unguided dialogue may increase the student’s engagement but may skirt around the right knowledge components. That is, the [[assistance dilemma]] surfaces as the degree of ''learner control'' (a term from the older educational literature) or ''student initiativ''e (a nearly synonymous term from the natural language dialogue literature).

== Explanation ==
If we view a short episode of interactive communication as a [[learning event space]], there could be three reasons why one treatment might be more effective than another:

(1) The learning event spaces might have different paths with different content. For instance, if one person contributes critical information that the other person lacks, then their joint learning event space has paths that are absent in the learning event space of the second person if that person were working solo. That is, the ''topology'' of one space might be better than the topology of the other.

(2) If the learning event spaces in the two conditions are the same, then the interactive communication treatment might cause the students to traverse different paths than the control students. That is, the ''path choices'' of one treatment might be better than the path choices of the other.

(3) If the learning event spaces are the same and the students take the same paths, they still might learn more in one condition than another because of the way that they traversed the path. For instance, having a partner observe the student as the student traverse a path might cause the student to be more attentive to details and to remember more. That is, the ''path effects'' might differ in the treatment vs. the control.

== Descendents ==

=== Collaboration ===
When and how does collaboration between peers can increase robust learning? Problem solving, example studying and many other activities can be done alone, in pairs, or in pairs with various kinds of assistance, such as collaboration scripts. From the standpoint of an individual learner, having a partner offers more [[assistance]] than working alone, and having a partner plus other scaffolding offer even more assistance. Thus, the [[Assistance Hypothesis]] predicts an interaction between various forms of peer collaboration and students' prior competence.

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

*[[Walker_A_Peer_Tutoring_Addition|Collaborative Extensions to the Cognitive Tutor Algebra: A Peer Tutoring Addition (Walker, McLaren, Koedinger, & Rummel)]]

*[[McLaren_et_al_-_Conceptual_Learning_in_Chemistry|Supporting Conceptual Learning in Chemistry through Collaboration Scripts and Adaptive, Online Support (McLaren, Rummel, Harrer, Spada, & Pinkwart)]]

=== Questioning ===
When and how can asking the student questions increase the student's robust learning? What kinds of questions are best?

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly, & Treacy, 2006)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008]])

*[[FrenchCulture|Understanding culture from film (Ogan, Aleven & Jones)]] [Also relevant to Refinement & Fluency, Explicit instruction and manipulations of attention & discrimination]

=== Tell vs. elicit ===
When a tutor knows that something needs to be said, she or he must decide whether to ''tell'' it to the tutee, try to ''elicit'' it from the tutee via a question or prompt, or just ''wait'' and hope that the tutee says it. Similarly, if a tutor knows that something needs to be done, the tutor can do it, elicit the action from the student or just wait. An instructional designer faces the same choices. For each thing that needs to be said or done in the instructional dialogue, should the tutor or the student be made responsible for it? For instance, should the tutoring system point out errors to the students or should the students detect their errors? In general, assistance is higher when the tutor does a portion of the instructional activity than when the student does it.

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Student_Uncertainty|Does Treating Student Uncertainty as a Learning Impasse Improve Learning in Spoken Dialogue Tutoring? (Forbes-Riley & Litman)]]

*[[The_Help_Tutor__Roll_Aleven_McLaren|Tutoring a meta-cognitive skill: Help-seeking (Roll, Aleven & McLaren)]] [Also in the Refinement & Fluency cluster, and relevant to Knowledge Component analysis]

*[[The self-correction of speech errors (McCormick, O’Neill & Siskin)]]

*[[Using Elaborated Explanations to Support Geometry Learning (Aleven & Butcher)]]

*[[Plateau_study|What is the optimal level of interaction during learning from problem solving? (Hausmann, van de Sande, & VanLehn, 2008)]]

[[Category:Cluster]]

Reflective Dialogues (Katz)

2008-04-15T21:28:37Z

Connelly: /* Connections */

== Do Reflective Dialogues that Explicitly Target the “What? How? and Why (not)?” Knowledge of Physics Problem Solving Promote Expert-like Planning Ability? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz, John Connelly, Donald Treacy
|-
| '''Study Start Date''' || 3/1/06
|-
| '''Study End Date''' || 6/30/07
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 67
|-
| '''Total Participant Hours''' || approx. 750 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
One of the main differences between experts and novices in physics is that experts are more adept at identifying relevant principles and generating solution plans before starting to solve a problem (Chi, Glaser & Rees, 1982; Dufresne, Gerace, Hardiman, & Mestre, 1992; Priest & Lindsay, 1992). We propose that this difference may be due to two reasons: (1) in traditional physics courses, students are not explicitly asked to plan or are given scaffolding to support planning during problem solving, and (2) many students lack the basic knowledge of physics concepts, principles, and procedures that is prerequisite for effective planning.

In this project, we tested the effectiveness of engaging students in reflective dialogues after they solve problems in [[Andes]] that explicitly target the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why (or why not) to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). The main hypothesis tested is that explicit training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, will be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

To test this hypothesis, we conducted a Physics LearnLab study in which we compared the effectiveness of an experimental version of Andes that engaged students in reflective “what? how? and why?” dialogues after physics problem solving ([[explicit instruction]] condition) with a control version that gave students additional problem-solving practice after they solved an Andes problem—e.g., by having them identify bugs in a sample student solution to a problem ([[implicit instruction]] condition). We predicted that students in the explicit condition would outperform students in the implicit condition with respect to gain score from pre-test to post-test and scores on course exams that targeted far transfer. Consistent with prior research, the dialogues promoted conceptual understanding of physics. However, measures of retention and transfer to problem-solving ability showed only a marginal effect of completing the reflective dialogues.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge—i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them—enhance students’ problem-solving ability?

=== Background and Significance ===
Because experts develop schemata largely through a good deal of practice with solving various types of problems, traditional introductory college physics courses take a similar approach. They are structured so that students are introduced to concepts, principles, and procedures in their text and through lectures, which they are then asked to apply to many problem-solving exercises. The hope is that, with extensive practice, students will integrate [[procedural]] and [[conceptual knowledge]] and develop expert-like schemata and planning skills. Unfortunately, many students don’t. In addition to exiting these courses with lingering naïve misconceptions (e.g., Halloun & Hestenes, 1985; McDermott, 1984), they continue to solve problems largely by manipulating equations until the desired quantity is isolated—that is, by means-end analysis—instead of by identifying relevant principles and generating solution plans before starting to solve a problem, as experts do (Dufresne et al., 1992; Larkin, McDermott, Simon, & Simon, 1980; Priest & Lindsay, 1992).

We refer to the traditional approach to physics instruction described in the preceding paragraph as [[implicit instruction]], because it encourages the inductive development of abstractions (concepts, principles, and schemata) through repeated exposure to instances, instead of by explicitly reifying these abstractions (O'Malley & Chamot, 1994). In response to the limitations of implicit approaches to physics instruction that neither ask students to plan nor scaffold them in doing it, several instructional scientists have proposed methods that explicitly engage students in planning exercises (Dufresne et al., 1992; Leonard, Dufresne, & Mestre, 1996; Mestre, Dufresne, Gerace, & Hardiman, 1993). These methods have met with modest success when tested mainly using high-achieving students (B or above in an introductory college physics course). We suggest that the main reason that many students, even high achievers, are unable to use explicit planning methods effectively is that they lack the basic concepts, principles, and procedures that are prerequisites for effective planning.

This project tested the effectiveness of engaging students in reflective dialogues after they solve problems in Andes (e.g., Gertner & VanLehn, 2000) which explicitly targeted the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). Explicit instruction on these knowledge components proved to be effective and more efficient than giving students repeated practice in solving different types of problems.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''Short-term retention''. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students within each course section were block-randomly assigned to one of two dialogue conditions: standard [[Knowledge Construction Dialogues]] (KCDs) or control with no KCDs. Students in the control condition were assigned an additional five Andes problems to attempt to better equate time on task.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of problems completed before the post-test was administered
* Number of dialogues that the student completed
* Grade point average (CQPR)
* College major
* Pre-test score

=== Hypothesis ===
The main hypothesis tested was that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

=== Findings ===
Analyses of pre- and post-test scores were more encouraging than for last year's study ([[Post-practice reflection (Katz)]]). After omitting scores from one student with a post-test duration of less than two minutes, we were left with treatment and control groups of equal size (''n'' = 33). We also re-classified two treatment subjects who did no dialogues as control subjects (effective treatment and control ''n''s = 31 and 35, respectively), although these subjects were not able to access the five extra control-group Andes problems. As we had hoped, ANOVA showed no significant differences between the effective treatment and control groups on pre-test score (respective ''M''s = 12.10 and 11.77; ''F'' < 1), but treatment subjects had higher mean post-test scores (17.97 vs. 15.57; ''F''(1, 64) = 4.89, ''p'' = .031), mean raw gain scores (5.87 vs. 3.80; ''F'' = 5.62, ''p'' = .021), and mean Estes gain scores (0.330 vs. 0.208; ''F'' = 6.74, ''p'' = .012) than did control subjects. In short, without regard to problem-solving practice, subjects who did KCDs did significantly better on the post-test than those who did not.

Student participation in both homework problem solving and dialogues was much improved over last year, with 21 of 34 control subjects (62%) and 23 of 33 treatment subjects (70%) finishing at least 80% of the assigned target problems (of 31 for control, 26 for treatment) prior to the post-test and with 25 treatment subjects (76%) finishing at least 80% of the 26 associated KCDs. However, participation was
still far from perfect; 7 treatment subjects (21%) completed half or fewer of the assigned KCDs on time, and 2 of them completed no target problems or KCDs on time. We again treated KCD completion and target problem completion as continuous independent variables in regression analyses. Regressing post-test score on pre-test score, QPA, number of KCDs completed, and number of target problems completed
(''R''2 = .52, ''F''(4, 61) = 16.70, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score, QPA, and KCD completion were statistically significant (''p''s < .001, .05, & .05, respectively); problem completion was ''ns'' (''p'' = .54). Therefore, across all subjects it was KCD completion, as opposed to target homework problem completion, that significantly predicted post-test performance.

To measure retention and transfer, we also analyzed scores from an hourly exam administered by one instructor to his two sections (''n''=47). All problems on this exam were quantitative and covered a subset of the units targeted during the intervention. Scores on this exam ranged from 176 to 382 (of 400), and were highly correlated with
pre- and post-test scores; ''r''s(45) = .54 and .71, ''p''s < .0001 and .00001. However,
ANOVAs showed no differences between groups (Fs < 1), and regression of subscores on QPA, KCDs completed, and target problems completed (''R''2 = .54, ''F''(3, 43) = 16.58,
''p'' < .00001) showed positive contributions of all factors but a significant effect of only QPA (''p'' < .00001). Therefore, student performance on the hourly exam was not significantly affected by either KCD completion or target problem completion.

In summary, student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed prior to the post-test. However, neither measure had a significant effect on student performance on an exam that focused on
problem-solving ability.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. More students took the learning-by doing path than in our prior study, perhaps because this year instructors ''required'' them to do so (i.e., this time there were negative consequences for avoiding them).

Analyses of retention and transfer to problem-solving ability did not reveal a significant effect of the reflective dialogues. We had hoped that the capstone dialogues that treatment students completed before selected problems, at the end of each unit, would support problem-solving ability. It is possible that the intervention consisted of too few capstone dialogues (only five) to observe any effect.

What seems to be the critical feature of post-practice reflection (PPR) in supporting conceptual understanding is the explicit instruction that it provides in domain knowledge. PPR may help students to fill in knowledge gaps, resolve misconceptions,
and abstract from the case at hand so that they are better prepared to engage in constructive activity (e.g., self-explanation) in future problems. Preliminary research comparing Andes (which encourages implicit learning of problem-solving strategies) with Pyrenees, a system that teaches problem-solving strategies explicitly, also
suggests an advantage for explicit instruction (VanLehn et al., 2004). Further research is needed to identify the mechanisms that drive learning from reflective dialogue, and to increase its potential to enhance problem-solving ability in addition to conceptual knowledge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Full paper accepted at AIED 2007:
**Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.

==== References ====
*Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), ''Advances in the Psychology of Human Intelligence, Vol. 1'' (pp. 7-75). Hillsdale, NJ: Erlbaum.
*Dufresne, R. J., Gerace, P., Hardiman, T., & Mestre, J. P. (1992). Constraining novices to perform expertlike analyses: Effects on schema acquisition. ''The Journal of the Learning Sciences, 2'' (3), 307-331.
*Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), ''ITS 2000: Proceedings of the 5th International Conference on Intelligent Tutoring Systems'' (pp. 133-142). Berlin: Springer-Verlag.
*Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. ''Science, 208'', 1335-1342.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*Mestre, J. P., Dufresne, R. J., Gerace, W. J., & Hardiman, P. T. (1993). Promoting skilled problem-solving behavior among beginning physics students. ''Journal of Research in Science Teaching, 30'', 303-317.
*O’Malley, M., and Chamot, A. (1994). ''The CALLA Handbook''. Reading, MA: Addison-Wesley.
*Priest, A. G., & Lindsay, R. O. (1992). New light on novice-expert differences in physics problem solving. ''British Journal of Psychology, 83'', 389-405.
*VanLehn, K., Bhembe, D., Chi, M., Lynch, C., Schulze, K., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2004). Implicit versus explicit learning of strategies in a nonprocedural cognitive skill. In J. C. Lester, R. M. Vicari, & F. Paraguacu, (Eds.), ''Intelligent Tutoring Systems: 7th International Conference'' (pp. 521-530). Berlin: Springer-Verlag.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans for January 2008 - December 2008:
* write journal article expanding AIED conference paper

[[Category:Study]]

Post-practice reflection (Katz)

2008-04-15T21:28:20Z

Connelly: /* Connections */

== Post-practice reflection in a first-year physics course: Does mixed-initiative interaction support robust learning better than tutor-led interaction or canned text? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PI''' || Sandra Katz
|-
| '''Other Contributors''' 
* '''Post-Docs:'''
| John Connelly
|-
| '''Study Start Date''' || 1/1/05
|-
| '''Study End Date''' || 12/31/05
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 123
|-
| '''Total Participant Hours''' || approx. 500 hrs
|-
| '''DataShop''' || No; Andes data still incompatible
|}

=== Abstract ===
We conducted an [[in vivo experiment]] within the PSLC Physics LearnLab to address two questions about reflection on recent problem-solving activity within [[Andes]]: (1) Does [[post-practice reflection]] support robust learning of physics—that is, students’ ability to [[transfer]] what they learned during instruction to novel situations? and (2) What is the preferred delivery mode for post-practice reflection in intelligent tutoring systems? We compared pre-test to post-test learning gains of students who were randomly assigned to one of four conditions that represent points on a scale of increasingly generative student activity: (1) a control group who solved a set of problems in Andes without any reflective activity, (2) a self-explanation group, who solved Andes problems, responded to several “[[reflection questions]]” (Lee & Hutchison, 1998) after each problem and received a canned, expert-generated explanation that they were prompted to compare with their response, (3) a “tutor-led interaction” group, who solved the same Andes problems and reflection questions (RQs) as the didactic group, but the questions were embedded in [[Knowledge Construction Dialogues]] (KCD’s) that guided students in generating a correct response, and (4) a “scaffolded mixed initiative KCD group,” which solved the same problems and responded to the same reflection questions as the other reflection groups, embedded in similar KCD’s as those presented to the “tutor-led interaction” group, with the added support of follow-up questioning menus that allowed students to take initiative.

Unfortunately, participation in the three experimental conditions was too low to compare the effectiveness of these treatments. However, a yoked pairs analysis revealed that the more reflection questions that students did, of any type, the better they did, with respect to both post-test scores and pre-test to post-test gain scores. Similarly, regression analysis revealed that the number of dialogues a student completed had a significant positive effect on post-test score, independent of the number of problems he or she completed. These findings provide empirical support for post-practice reflection in an ITS that is a central component of a course.

=== Glossary ===

* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research questions ===

Do reflection questions after physics problem solving support robust learning? Is more interactive reflection, between the automated tutor and student, more effective in supporting robust learning than less interactive reflection?

=== Background and Significance ===

Despite the well-established role that post-practice, reflective dialogue plays in apprentice-style learning (Collins, Brown, & Newman, 1989), few studies have been conducted to test their effectiveness. Hence, there is little empirical support for incorporating post-practice reflection in courses led by a human teacher or within intelligent tutoring systems (ITSs). Nor is there much guidance on how to implement reflection effectively, especially within the constraints of natural-language understanding and generation capabilities.

Research by Lee and Hutchison (1998) focused on the effectiveness of reflection questions posed after students studied worked examples on balancing chemistry equations in a computer-based learning environment. This study found that reflection questions enhanced students’ problem-solving ability. However, the role of reflection in promoting conceptual understanding was not addressed; nor did the study investigate whether reflection questions after problem-solving exercises (as opposed to example studying) support learning.

Two laboratory studies conducted by Katz and her colleagues addressed these questions. In a longitudinal analysis of student performance on avionics tasks in Sherlock 2 (e.g., Katz et al., 1998), Katz, O’Donnell and Kay (2000) found that discussions that took place between an avionics expert and two students were more effective in resolving misconceptions when they were distributed across problem solving and debrief than were discussions that took place during problem solving alone. However, due to constraints inherent in the research setting, there was no control group in this study—that is, avionics trainees who did not experience debrief led by a domain expert—and no instrument to measure performance gains. A follow-up study by Katz, Allbritton, & Connelly (2003) addressed these limitations in a different domain, first-year college physics. Forty-six students taking college physics solved problems in Andes in one of three conditions: with no reflection questions after problem solving (control group), with reflection questions discussed with human tutors, or with the same reflection questions followed by canned feedback (without a human tutor). A comparison of pre-test and post-test scores was conducted to measure learning gains. The main result was that students learned more with reflection questions than without, but the two conditions with reflection questions (canned feedback and human tutoring) did not differ significantly.

The current study is significant because it validated the results of these laboratory studies in an actual classroom setting. Specifically, it showed that post-practice reflection supports learning, as measured by pre-test to post-test learning gain scores. Due to low participation, however, this experiment did not shed light on the question of which modality of reflection is most effective.

=== Dependent Variables ===

* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near transfer ([[normal post-test]]) and far [[transfer]] (robust learning) items.

* ''Short-term retention''. Performance on course exam that covered target topic (work and energy), 1-2 weeks after the intervention.

* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students were block-randomly assigned to one of four dialogue conditions:
* control (Ctrl), with no relection questions and KCDs;
* canned text response (CTR), with free-response KCDs followed by pre-scripted expert responses; Studnet's responses were not given a positive/negative feedback.
* standard KCDs (KCD), with shorter-answer questions and tutor responses that indicate what parts of the student's response was incomplete or incorrect and try to elicit the correct aspects; and
* limited mixed-initiative (MIX), same as the standard KCDs, but with on-demand hypertext question-and-answer links.

However, there were no significant differences between categorical condition assignments, even after reclassifying as Ctrl subjects those in the three treatment conditions who saw no KCDs and after collapsing the KCD and MIX conditions (due to minimal usage of mixed-initiative links, for the most part the two conditions were equivalent). We therefore treated KCD and problem completion as continuous variables.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

*Number of problems completed before the post-test was administered
*Number of reflection questions that the student completed
*CQPR—grade point average
*College major grouping
*Pre-test score

=== Hypotheses ===

*Because responding to reflection questions is a form of “active learning,” students in any of the three experimental conditions (self-explanation of canned text, standard KCD’s, tutor-led KCDs) should outperform students in the control (no-reflection) condition. This hypothesis was supported.
*The more interactive the reflection modality the better, so mixed-initiative > standard KCD > self-explanation of canned text. We were unable to test this hypothesis, due to low participation.

=== Findings ===

*''Gain scores summary'':
**There were no significant differences in mean gain score (or in pre-test and post-test scores) by condition (4 conditions; 3 treatment, 1 control) or by collapsed effective condition (Ctrl, CTR, KCD).
**Yoked pairs analysis, comparing students from the same major groups with identical pre-tests scores (and minimal CQPR disparity) who completed no reflection questions with students who completed five or more questions (''n'' = 19 pairs), showed that “treated” subjects tended to out-gain “untreated” subjects.
*''Regression analysis summary'':
**The number of reflection questions completed had a significant positive effect on post-test scores.
*''Exam score summary'':
**For the final exam, there was no significant impact of the number of reflection questions completed.
**For one course section’s hourly exam on work and energy, there was a significant, positive impact of the number of reflection questions completed, but only when CQPR was dropped from the model. For the other section’s hourly exam, “treated” subjects significantly outperformed “untreated” subjects, despite being outnumbered 3 to 1.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. We are unable to determine why students chose not to take the learning-by doing path. Perhaps doing the reflection questions was too difficult for some students, but we suspect that this was not the case. A more likely explanation is that students did not have the time to complete these questions, and there were no negative consequences for avoiding them. In the follow-up study that we are currently running, students are required to complete the reflection questions in order to get credit for completing a problem.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Poster presentation at the 2006 American Association for Physics Teachers (AAPT) Conference
* Full-paper accepted at AIED 2007

==== References ====
*Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. In L. B. Resnick (Ed.), ''Knowing, learning and instruction: Essays in honor of Robert Glaser'' (pp. 543-494). Hillsdale, NJ: Erlbaum.
*Lee, A. Y., & Hutchison, L. (1998). Improving learning from examples through reflection. ''Journal of Experimental Psychology: Applied, 4'' (3), 187-210.
*Katz, S., & Allbritton, D., & Connelly, J. (2003). Going beyond the problem given: How human tutors use post-solution discussions to support transfer. ''International Journal of Artificial Intelligence and Education, 13'' (1), 79-116.
*Katz, S., Lesgold, A., Hughes, E., Peters, D., Eggan, G., Gordin, M., & Greenberg, L. (1998). Sherlock II: An intelligent tutoring system built upon the LRDC tutor framework. In C. P. Bloom & R. B. Loftin (Eds.), ''Facilitating the development and use of interactive learning environments'' (pp. 227-258). Mahwah, NJ: Erlbaum.
*Katz, S., O’Donnell, G., & Kay, H. (2000). An approach to analyzing the role and structure of reflective dialogue. ''International Journal of Artificial Intelligence and Education, 11'', 320-343.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T21:27:44Z

Connelly: /* Connections */

Interactive Communication

2008-04-15T21:26:07Z

Connelly: /* Questioning */

= The PSLC Interactive Communication cluster =

== Abstract ==
The studies in the Interactive Communication deal primarily with learning environments where there are two interacting, communicating agents, one of which is the student. The other [[agent]] is typically a second student, a human tutor or a tutoring system. They communicate, either in a natural language or a formal language, such as mathematical expression or menus. We are trying to find out why such instructional, dyadic, interactive communication is sometimes highly effective and sometimes less effective. Sometimes we study highly constrained forms of communication in order to vary isolated aspects, and sometimes we compare whole forms of communciation. Our hypothesis is simply that interactive communication is effective if it guides students to attend to the right [[knowledge components]].

 <center>[[Image:ic-theory.jpg]]</center>

== Background and Significance ==
Although instructional dialogue has been studied in classrooms (e.g., Lave & Wenger, 1991; Leinhardt, 1990) and workplaces (e.g., Hutchins, 1995; Nunes, Schliemann & Carraher, 1993), we are focusing on more tractable albeit still complex situations: ''dyadic'' instructional dialogues, namely dialogues between: (a) a human tutor and a human student, (b) two human students, or (c) A computer tutor and a human student. Moreover, the dialogue are task-oriented (Grosz & Sidner, 1986) in that the participants are working together on a task rather than simply conversing with no shared goals or with opposing goals.

Early studies focused on the structure of dyadic instructional dialogue (e.g., Fox, 1993; Graesser, Person & Magliano, 1995; MacArthur, Stasz, & Zmuidzinas, 1990). When later studies compared the learning that occurred during dialogue vs. less interactive instruction (e.g., VanLehn, Graesser et al., 2007; Katz, Connelly & Allbritton, 2003; Evens & Michael, 2006; Cohen, Kulik & Kulik, 1982), they found surprisingly mixed results. Only 60% of the studies showed that interactive communication caused larger learning gains than less interactive instruction.

The interactive communication cluster is undertaking the next step in this important line of research by investigating when different types of interactive communication are effective and why. Sometimes we compare highly constrained forms of communication in order to vary isolated aspects, and sometimes we compare constrained interactive communication to passive communication (e.g., reading).

== Glossary ==
See [[:Category:Interactive Communication|Interactive Communication Glossary]]

== Research question ==
What properties of interactive communication promote robust learning?

== Independent variables ==
The independent variables (also called Treatment Variables) of the IC cluster appear as column headers in the matrix above. They are listed here with links to their glossary entries.

* [[Collaboration]]

* [[Vicarious learning]]

* [[Collaboration scripts]]

* [[Deep/Reflection questions]]

* [[Instructional explanation]]

* [[Prompted Self-explanation]]

* [[Tutoring feedback]]

* [[Error correction support]]

== Dependent variables ==
Measures of normal and robust learning.

== Hypothesis ==
Our central hypothesis is just a special case of the [[Knowledge component hypothesis]]: interactive communication is effective if it guides students to attend to the right [[knowledge components]]. The key words here are “guide” and “attend” because they may oppose each other. A dialogue that strongly guides the student may also cause the student to disengage and thus not attend to the knowledge component even if the student’s dialogue partner mentions them. On the other hand, an unguided dialogue may increase the student’s engagement but may skirt around the right knowledge components. That is, the [[assistance dilemma]] surfaces as the degree of ''learner control'' (a term from the older educational literature) or ''student initiativ''e (a nearly synonymous term from the natural language dialogue literature).

== Explanation ==
If we view a short episode of interactive communication as a [[learning event space]], there could be three reasons why one treatment might be more effective than another:

(1) The learning event spaces might have different paths with different content. For instance, if one person contributes critical information that the other person lacks, then their joint learning event space has paths that are absent in the learning event space of the second person if that person were working solo. That is, the ''topology'' of one space might be better than the topology of the other.

(2) If the learning event spaces in the two conditions are the same, then the interactive communication treatment might cause the students to traverse different paths than the control students. That is, the ''path choices'' of one treatment might be better than the path choices of the other.

(3) If the learning event spaces are the same and the students take the same paths, they still might learn more in one condition than another because of the way that they traversed the path. For instance, having a partner observe the student as the student traverse a path might cause the student to be more attentive to details and to remember more. That is, the ''path effects'' might differ in the treatment vs. the control.

== Descendents ==

=== Collaboration ===
When and how does collaboration between peers can increase robust learning? Problem solving, example studying and many other activities can be done alone, in pairs, or in pairs with various kinds of assistance, such as collaboration scripts. From the standpoint of an individual learner, having a partner offers more [[assistance]] than working alone, and having a partner plus other scaffolding offer even more assistance. Thus, the [[Assistance Hypothesis]] predicts an interaction between various forms of peer collaboration and students' prior competence.

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

*[[Walker_A_Peer_Tutoring_Addition|Collaborative Extensions to the Cognitive Tutor Algebra: A Peer Tutoring Addition (Walker, McLaren, Koedinger, & Rummel)]]

*[[McLaren_et_al_-_Conceptual_Learning_in_Chemistry|Supporting Conceptual Learning in Chemistry through Collaboration Scripts and Adaptive, Online Support (McLaren, Rummel, Harrer, Spada, & Pinkwart)]]

=== Questioning ===
When and how can asking the student questions increase the student's robust learning? What kinds of questions are best?

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2007)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008]])

*[[FrenchCulture|Understanding culture from film (Ogan, Aleven & Jones)]] [Also relevant to Refinement & Fluency, Explicit instruction and manipulations of attention & discrimination]

=== Tell vs. elicit ===
When a tutor knows that something needs to be said, she or he must decide whether to ''tell'' it to the tutee, try to ''elicit'' it from the tutee via a question or prompt, or just ''wait'' and hope that the tutee says it. Similarly, if a tutor knows that something needs to be done, the tutor can do it, elicit the action from the student or just wait. An instructional designer faces the same choices. For each thing that needs to be said or done in the instructional dialogue, should the tutor or the student be made responsible for it? For instance, should the tutoring system point out errors to the students or should the students detect their errors? In general, assistance is higher when the tutor does a portion of the instructional activity than when the student does it.

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Student_Uncertainty|Does Treating Student Uncertainty as a Learning Impasse Improve Learning in Spoken Dialogue Tutoring? (Forbes-Riley & Litman)]]

*[[The_Help_Tutor__Roll_Aleven_McLaren|Tutoring a meta-cognitive skill: Help-seeking (Roll, Aleven & McLaren)]] [Also in the Refinement & Fluency cluster, and relevant to Knowledge Component analysis]

*[[The self-correction of speech errors (McCormick, O’Neill & Siskin)]]

*[[Using Elaborated Explanations to Support Geometry Learning (Aleven & Butcher)]]

*[[Plateau_study|What is the optimal level of interaction during learning from problem solving? (Hausmann, van de Sande, & VanLehn, 2008)]]

[[Category:Cluster]]

Reflective Dialogues (Katz)

2008-04-15T21:24:23Z

Connelly: /* Connections */

== Do Reflective Dialogues that Explicitly Target the “What? How? and Why (not)?” Knowledge of Physics Problem Solving Promote Expert-like Planning Ability? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz, John Connelly, Donald Treacy
|-
| '''Study Start Date''' || 3/1/06
|-
| '''Study End Date''' || 6/30/07
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 67
|-
| '''Total Participant Hours''' || approx. 750 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
One of the main differences between experts and novices in physics is that experts are more adept at identifying relevant principles and generating solution plans before starting to solve a problem (Chi, Glaser & Rees, 1982; Dufresne, Gerace, Hardiman, & Mestre, 1992; Priest & Lindsay, 1992). We propose that this difference may be due to two reasons: (1) in traditional physics courses, students are not explicitly asked to plan or are given scaffolding to support planning during problem solving, and (2) many students lack the basic knowledge of physics concepts, principles, and procedures that is prerequisite for effective planning.

In this project, we tested the effectiveness of engaging students in reflective dialogues after they solve problems in [[Andes]] that explicitly target the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why (or why not) to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). The main hypothesis tested is that explicit training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, will be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

To test this hypothesis, we conducted a Physics LearnLab study in which we compared the effectiveness of an experimental version of Andes that engaged students in reflective “what? how? and why?” dialogues after physics problem solving ([[explicit instruction]] condition) with a control version that gave students additional problem-solving practice after they solved an Andes problem—e.g., by having them identify bugs in a sample student solution to a problem ([[implicit instruction]] condition). We predicted that students in the explicit condition would outperform students in the implicit condition with respect to gain score from pre-test to post-test and scores on course exams that targeted far transfer. Consistent with prior research, the dialogues promoted conceptual understanding of physics. However, measures of retention and transfer to problem-solving ability showed only a marginal effect of completing the reflective dialogues.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge—i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them—enhance students’ problem-solving ability?

=== Background and Significance ===
Because experts develop schemata largely through a good deal of practice with solving various types of problems, traditional introductory college physics courses take a similar approach. They are structured so that students are introduced to concepts, principles, and procedures in their text and through lectures, which they are then asked to apply to many problem-solving exercises. The hope is that, with extensive practice, students will integrate [[procedural]] and [[conceptual knowledge]] and develop expert-like schemata and planning skills. Unfortunately, many students don’t. In addition to exiting these courses with lingering naïve misconceptions (e.g., Halloun & Hestenes, 1985; McDermott, 1984), they continue to solve problems largely by manipulating equations until the desired quantity is isolated—that is, by means-end analysis—instead of by identifying relevant principles and generating solution plans before starting to solve a problem, as experts do (Dufresne et al., 1992; Larkin, McDermott, Simon, & Simon, 1980; Priest & Lindsay, 1992).

We refer to the traditional approach to physics instruction described in the preceding paragraph as [[implicit instruction]], because it encourages the inductive development of abstractions (concepts, principles, and schemata) through repeated exposure to instances, instead of by explicitly reifying these abstractions (O'Malley & Chamot, 1994). In response to the limitations of implicit approaches to physics instruction that neither ask students to plan nor scaffold them in doing it, several instructional scientists have proposed methods that explicitly engage students in planning exercises (Dufresne et al., 1992; Leonard, Dufresne, & Mestre, 1996; Mestre, Dufresne, Gerace, & Hardiman, 1993). These methods have met with modest success when tested mainly using high-achieving students (B or above in an introductory college physics course). We suggest that the main reason that many students, even high achievers, are unable to use explicit planning methods effectively is that they lack the basic concepts, principles, and procedures that are prerequisites for effective planning.

This project tested the effectiveness of engaging students in reflective dialogues after they solve problems in Andes (e.g., Gertner & VanLehn, 2000) which explicitly targeted the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). Explicit instruction on these knowledge components proved to be effective and more efficient than giving students repeated practice in solving different types of problems.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''Short-term retention''. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students within each course section were block-randomly assigned to one of two dialogue conditions: standard [[Knowledge Construction Dialogues]] (KCDs) or control with no KCDs. Students in the control condition were assigned an additional five Andes problems to attempt to better equate time on task.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of problems completed before the post-test was administered
* Number of dialogues that the student completed
* Grade point average (CQPR)
* College major
* Pre-test score

=== Hypothesis ===
The main hypothesis tested was that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

=== Findings ===
Analyses of pre- and post-test scores were more encouraging than for last year's study ([[Post-practice reflection (Katz)]]). After omitting scores from one student with a post-test duration of less than two minutes, we were left with treatment and control groups of equal size (''n'' = 33). We also re-classified two treatment subjects who did no dialogues as control subjects (effective treatment and control ''n''s = 31 and 35, respectively), although these subjects were not able to access the five extra control-group Andes problems. As we had hoped, ANOVA showed no significant differences between the effective treatment and control groups on pre-test score (respective ''M''s = 12.10 and 11.77; ''F'' < 1), but treatment subjects had higher mean post-test scores (17.97 vs. 15.57; ''F''(1, 64) = 4.89, ''p'' = .031), mean raw gain scores (5.87 vs. 3.80; ''F'' = 5.62, ''p'' = .021), and mean Estes gain scores (0.330 vs. 0.208; ''F'' = 6.74, ''p'' = .012) than did control subjects. In short, without regard to problem-solving practice, subjects who did KCDs did significantly better on the post-test than those who did not.

Student participation in both homework problem solving and dialogues was much improved over last year, with 21 of 34 control subjects (62%) and 23 of 33 treatment subjects (70%) finishing at least 80% of the assigned target problems (of 31 for control, 26 for treatment) prior to the post-test and with 25 treatment subjects (76%) finishing at least 80% of the 26 associated KCDs. However, participation was
still far from perfect; 7 treatment subjects (21%) completed half or fewer of the assigned KCDs on time, and 2 of them completed no target problems or KCDs on time. We again treated KCD completion and target problem completion as continuous independent variables in regression analyses. Regressing post-test score on pre-test score, QPA, number of KCDs completed, and number of target problems completed
(''R''2 = .52, ''F''(4, 61) = 16.70, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score, QPA, and KCD completion were statistically significant (''p''s < .001, .05, & .05, respectively); problem completion was ''ns'' (''p'' = .54). Therefore, across all subjects it was KCD completion, as opposed to target homework problem completion, that significantly predicted post-test performance.

To measure retention and transfer, we also analyzed scores from an hourly exam administered by one instructor to his two sections (''n''=47). All problems on this exam were quantitative and covered a subset of the units targeted during the intervention. Scores on this exam ranged from 176 to 382 (of 400), and were highly correlated with
pre- and post-test scores; ''r''s(45) = .54 and .71, ''p''s < .0001 and .00001. However,
ANOVAs showed no differences between groups (Fs < 1), and regression of subscores on QPA, KCDs completed, and target problems completed (''R''2 = .54, ''F''(3, 43) = 16.58,
''p'' < .00001) showed positive contributions of all factors but a significant effect of only QPA (''p'' < .00001). Therefore, student performance on the hourly exam was not significantly affected by either KCD completion or target problem completion.

In summary, student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed prior to the post-test. However, neither measure had a significant effect on student performance on an exam that focused on
problem-solving ability.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. More students took the learning-by doing path than in our prior study, perhaps because this year instructors ''required'' them to do so (i.e., this time there were negative consequences for avoiding them).

Analyses of retention and transfer to problem-solving ability did not reveal a significant effect of the reflective dialogues. We had hoped that the capstone dialogues that treatment students completed before selected problems, at the end of each unit, would support problem-solving ability. It is possible that the intervention consisted of too few capstone dialogues (only five) to observe any effect.

What seems to be the critical feature of post-practice reflection (PPR) in supporting conceptual understanding is the explicit instruction that it provides in domain knowledge. PPR may help students to fill in knowledge gaps, resolve misconceptions,
and abstract from the case at hand so that they are better prepared to engage in constructive activity (e.g., self-explanation) in future problems. Preliminary research comparing Andes (which encourages implicit learning of problem-solving strategies) with Pyrenees, a system that teaches problem-solving strategies explicitly, also
suggests an advantage for explicit instruction (VanLehn et al., 2004). Further research is needed to identify the mechanisms that drive learning from reflective dialogue, and to increase its potential to enhance problem-solving ability in addition to conceptual knowledge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Full paper accepted at AIED 2007:
**Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.

==== References ====
*Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), ''Advances in the Psychology of Human Intelligence, Vol. 1'' (pp. 7-75). Hillsdale, NJ: Erlbaum.
*Dufresne, R. J., Gerace, P., Hardiman, T., & Mestre, J. P. (1992). Constraining novices to perform expertlike analyses: Effects on schema acquisition. ''The Journal of the Learning Sciences, 2'' (3), 307-331.
*Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), ''ITS 2000: Proceedings of the 5th International Conference on Intelligent Tutoring Systems'' (pp. 133-142). Berlin: Springer-Verlag.
*Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. ''Science, 208'', 1335-1342.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*Mestre, J. P., Dufresne, R. J., Gerace, W. J., & Hardiman, P. T. (1993). Promoting skilled problem-solving behavior among beginning physics students. ''Journal of Research in Science Teaching, 30'', 303-317.
*O’Malley, M., and Chamot, A. (1994). ''The CALLA Handbook''. Reading, MA: Addison-Wesley.
*Priest, A. G., & Lindsay, R. O. (1992). New light on novice-expert differences in physics problem solving. ''British Journal of Psychology, 83'', 389-405.
*VanLehn, K., Bhembe, D., Chi, M., Lynch, C., Schulze, K., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2004). Implicit versus explicit learning of strategies in a nonprocedural cognitive skill. In J. C. Lester, R. M. Vicari, & F. Paraguacu, (Eds.), ''Intelligent Tutoring Systems: 7th International Conference'' (pp. 521-530). Berlin: Springer-Verlag.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2006)]]
* [[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Reflective Dialogue Support (Katz & Connelly, 2008)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans for January 2008 - December 2008:
* write journal article expanding AIED conference paper

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T21:22:37Z

Connelly: /* Connections */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T21:21:50Z

Connelly: /* Findings */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T21:20:41Z

Connelly:

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T21:19:17Z

Connelly:

Reflective Dialogues (Katz)

2008-04-15T21:07:36Z

Connelly: /* Further Information */

== Do Reflective Dialogues that Explicitly Target the “What? How? and Why (not)?” Knowledge of Physics Problem Solving Promote Expert-like Planning Ability? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz, John Connelly, Donald Treacy
|-
| '''Study Start Date''' || 3/1/06
|-
| '''Study End Date''' || 6/30/07
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 67
|-
| '''Total Participant Hours''' || approx. 750 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
One of the main differences between experts and novices in physics is that experts are more adept at identifying relevant principles and generating solution plans before starting to solve a problem (Chi, Glaser & Rees, 1982; Dufresne, Gerace, Hardiman, & Mestre, 1992; Priest & Lindsay, 1992). We propose that this difference may be due to two reasons: (1) in traditional physics courses, students are not explicitly asked to plan or are given scaffolding to support planning during problem solving, and (2) many students lack the basic knowledge of physics concepts, principles, and procedures that is prerequisite for effective planning.

In this project, we tested the effectiveness of engaging students in reflective dialogues after they solve problems in [[Andes]] that explicitly target the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why (or why not) to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). The main hypothesis tested is that explicit training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, will be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

To test this hypothesis, we conducted a Physics LearnLab study in which we compared the effectiveness of an experimental version of Andes that engaged students in reflective “what? how? and why?” dialogues after physics problem solving ([[explicit instruction]] condition) with a control version that gave students additional problem-solving practice after they solved an Andes problem—e.g., by having them identify bugs in a sample student solution to a problem ([[implicit instruction]] condition). We predicted that students in the explicit condition would outperform students in the implicit condition with respect to gain score from pre-test to post-test and scores on course exams that targeted far transfer. Consistent with prior research, the dialogues promoted conceptual understanding of physics. However, measures of retention and transfer to problem-solving ability showed only a marginal effect of completing the reflective dialogues.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge—i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them—enhance students’ problem-solving ability?

=== Background and Significance ===
Because experts develop schemata largely through a good deal of practice with solving various types of problems, traditional introductory college physics courses take a similar approach. They are structured so that students are introduced to concepts, principles, and procedures in their text and through lectures, which they are then asked to apply to many problem-solving exercises. The hope is that, with extensive practice, students will integrate [[procedural]] and [[conceptual knowledge]] and develop expert-like schemata and planning skills. Unfortunately, many students don’t. In addition to exiting these courses with lingering naïve misconceptions (e.g., Halloun & Hestenes, 1985; McDermott, 1984), they continue to solve problems largely by manipulating equations until the desired quantity is isolated—that is, by means-end analysis—instead of by identifying relevant principles and generating solution plans before starting to solve a problem, as experts do (Dufresne et al., 1992; Larkin, McDermott, Simon, & Simon, 1980; Priest & Lindsay, 1992).

We refer to the traditional approach to physics instruction described in the preceding paragraph as [[implicit instruction]], because it encourages the inductive development of abstractions (concepts, principles, and schemata) through repeated exposure to instances, instead of by explicitly reifying these abstractions (O'Malley & Chamot, 1994). In response to the limitations of implicit approaches to physics instruction that neither ask students to plan nor scaffold them in doing it, several instructional scientists have proposed methods that explicitly engage students in planning exercises (Dufresne et al., 1992; Leonard, Dufresne, & Mestre, 1996; Mestre, Dufresne, Gerace, & Hardiman, 1993). These methods have met with modest success when tested mainly using high-achieving students (B or above in an introductory college physics course). We suggest that the main reason that many students, even high achievers, are unable to use explicit planning methods effectively is that they lack the basic concepts, principles, and procedures that are prerequisites for effective planning.

This project tested the effectiveness of engaging students in reflective dialogues after they solve problems in Andes (e.g., Gertner & VanLehn, 2000) which explicitly targeted the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). Explicit instruction on these knowledge components proved to be effective and more efficient than giving students repeated practice in solving different types of problems.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''Short-term retention''. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students within each course section were block-randomly assigned to one of two dialogue conditions: standard [[Knowledge Construction Dialogues]] (KCDs) or control with no KCDs. Students in the control condition were assigned an additional five Andes problems to attempt to better equate time on task.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of problems completed before the post-test was administered
* Number of dialogues that the student completed
* Grade point average (CQPR)
* College major
* Pre-test score

=== Hypothesis ===
The main hypothesis tested was that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

=== Findings ===
Analyses of pre- and post-test scores were more encouraging than for last year's study ([[Post-practice reflection (Katz)]]). After omitting scores from one student with a post-test duration of less than two minutes, we were left with treatment and control groups of equal size (''n'' = 33). We also re-classified two treatment subjects who did no dialogues as control subjects (effective treatment and control ''n''s = 31 and 35, respectively), although these subjects were not able to access the five extra control-group Andes problems. As we had hoped, ANOVA showed no significant differences between the effective treatment and control groups on pre-test score (respective ''M''s = 12.10 and 11.77; ''F'' < 1), but treatment subjects had higher mean post-test scores (17.97 vs. 15.57; ''F''(1, 64) = 4.89, ''p'' = .031), mean raw gain scores (5.87 vs. 3.80; ''F'' = 5.62, ''p'' = .021), and mean Estes gain scores (0.330 vs. 0.208; ''F'' = 6.74, ''p'' = .012) than did control subjects. In short, without regard to problem-solving practice, subjects who did KCDs did significantly better on the post-test than those who did not.

Student participation in both homework problem solving and dialogues was much improved over last year, with 21 of 34 control subjects (62%) and 23 of 33 treatment subjects (70%) finishing at least 80% of the assigned target problems (of 31 for control, 26 for treatment) prior to the post-test and with 25 treatment subjects (76%) finishing at least 80% of the 26 associated KCDs. However, participation was
still far from perfect; 7 treatment subjects (21%) completed half or fewer of the assigned KCDs on time, and 2 of them completed no target problems or KCDs on time. We again treated KCD completion and target problem completion as continuous independent variables in regression analyses. Regressing post-test score on pre-test score, QPA, number of KCDs completed, and number of target problems completed
(''R''2 = .52, ''F''(4, 61) = 16.70, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score, QPA, and KCD completion were statistically significant (''p''s < .001, .05, & .05, respectively); problem completion was ''ns'' (''p'' = .54). Therefore, across all subjects it was KCD completion, as opposed to target homework problem completion, that significantly predicted post-test performance.

To measure retention and transfer, we also analyzed scores from an hourly exam administered by one instructor to his two sections (''n''=47). All problems on this exam were quantitative and covered a subset of the units targeted during the intervention. Scores on this exam ranged from 176 to 382 (of 400), and were highly correlated with
pre- and post-test scores; ''r''s(45) = .54 and .71, ''p''s < .0001 and .00001. However,
ANOVAs showed no differences between groups (Fs < 1), and regression of subscores on QPA, KCDs completed, and target problems completed (''R''2 = .54, ''F''(3, 43) = 16.58,
''p'' < .00001) showed positive contributions of all factors but a significant effect of only QPA (''p'' < .00001). Therefore, student performance on the hourly exam was not significantly affected by either KCD completion or target problem completion.

In summary, student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed prior to the post-test. However, neither measure had a significant effect on student performance on an exam that focused on
problem-solving ability.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. More students took the learning-by doing path than in our prior study, perhaps because this year instructors ''required'' them to do so (i.e., this time there were negative consequences for avoiding them).

Analyses of retention and transfer to problem-solving ability did not reveal a significant effect of the reflective dialogues. We had hoped that the capstone dialogues that treatment students completed before selected problems, at the end of each unit, would support problem-solving ability. It is possible that the intervention consisted of too few capstone dialogues (only five) to observe any effect.

What seems to be the critical feature of post-practice reflection (PPR) in supporting conceptual understanding is the explicit instruction that it provides in domain knowledge. PPR may help students to fill in knowledge gaps, resolve misconceptions,
and abstract from the case at hand so that they are better prepared to engage in constructive activity (e.g., self-explanation) in future problems. Preliminary research comparing Andes (which encourages implicit learning of problem-solving strategies) with Pyrenees, a system that teaches problem-solving strategies explicitly, also
suggests an advantage for explicit instruction (VanLehn et al., 2004). Further research is needed to identify the mechanisms that drive learning from reflective dialogue, and to increase its potential to enhance problem-solving ability in addition to conceptual knowledge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Full paper accepted at AIED 2007:
**Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), ''Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work'' (pp. 425-432). Amsterdam: IOS Press.

==== References ====
*Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), ''Advances in the Psychology of Human Intelligence, Vol. 1'' (pp. 7-75). Hillsdale, NJ: Erlbaum.
*Dufresne, R. J., Gerace, P., Hardiman, T., & Mestre, J. P. (1992). Constraining novices to perform expertlike analyses: Effects on schema acquisition. ''The Journal of the Learning Sciences, 2'' (3), 307-331.
*Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), ''ITS 2000: Proceedings of the 5th International Conference on Intelligent Tutoring Systems'' (pp. 133-142). Berlin: Springer-Verlag.
*Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. ''Science, 208'', 1335-1342.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*Mestre, J. P., Dufresne, R. J., Gerace, W. J., & Hardiman, P. T. (1993). Promoting skilled problem-solving behavior among beginning physics students. ''Journal of Research in Science Teaching, 30'', 303-317.
*O’Malley, M., and Chamot, A. (1994). ''The CALLA Handbook''. Reading, MA: Addison-Wesley.
*Priest, A. G., & Lindsay, R. O. (1992). New light on novice-expert differences in physics problem solving. ''British Journal of Psychology, 83'', 389-405.
*VanLehn, K., Bhembe, D., Chi, M., Lynch, C., Schulze, K., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2004). Implicit versus explicit learning of strategies in a nonprocedural cognitive skill. In J. C. Lester, R. M. Vicari, & F. Paraguacu, (Eds.), ''Intelligent Tutoring Systems: 7th International Conference'' (pp. 521-530). Berlin: Springer-Verlag.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans for January 2008 - December 2008:
* write journal article expanding AIED conference paper

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T20:49:16Z

Connelly:

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T20:40:43Z

Connelly:

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-15T20:17:52Z

Connelly:

Reflective Dialogues (Katz)

2008-04-15T19:38:01Z

Connelly: /* Background and Significance */

== Do Reflective Dialogues that Explicitly Target the “What? How? and Why (not)?” Knowledge of Physics Problem Solving Promote Expert-like Planning Ability? ==
Sandra Katz

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Sandra Katz, John Connelly, Donald Treacy
|-
| '''Study Start Date''' || 3/1/06
|-
| '''Study End Date''' || 6/30/07
|-
| '''LearnLab Site''' || USNA
|-
| '''LearnLab Course''' || General Physics I
|-
| '''Number of Students''' || ''N'' = 67
|-
| '''Total Participant Hours''' || approx. 750 hrs
|-
| '''DataShop''' || Yes
|}

=== Abstract ===
One of the main differences between experts and novices in physics is that experts are more adept at identifying relevant principles and generating solution plans before starting to solve a problem (Chi, Glaser & Rees, 1982; Dufresne, Gerace, Hardiman, & Mestre, 1992; Priest & Lindsay, 1992). We propose that this difference may be due to two reasons: (1) in traditional physics courses, students are not explicitly asked to plan or are given scaffolding to support planning during problem solving, and (2) many students lack the basic knowledge of physics concepts, principles, and procedures that is prerequisite for effective planning.

In this project, we tested the effectiveness of engaging students in reflective dialogues after they solve problems in [[Andes]] that explicitly target the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why (or why not) to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). The main hypothesis tested is that explicit training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, will be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

To test this hypothesis, we conducted a Physics LearnLab study in which we compared the effectiveness of an experimental version of Andes that engaged students in reflective “what? how? and why?” dialogues after physics problem solving ([[explicit instruction]] condition) with a control version that gave students additional problem-solving practice after they solved an Andes problem—e.g., by having them identify bugs in a sample student solution to a problem ([[implicit instruction]] condition). We predicted that students in the explicit condition would outperform students in the implicit condition with respect to gain score from pre-test to post-test and scores on course exams that targeted far transfer. Consistent with prior research, the dialogues promoted conceptual understanding of physics. However, measures of retention and transfer to problem-solving ability showed only a marginal effect of completing the reflective dialogues.

=== Glossary ===
* [[Post-practice reflection]]
* [[Reflection questions]]
* [[Knowledge Construction Dialogues]] (KCDs)

=== Research question ===
Does explicit training in the three main components of problem-solving knowledge—i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them—enhance students’ problem-solving ability?

=== Background and Significance ===
Because experts develop schemata largely through a good deal of practice with solving various types of problems, traditional introductory college physics courses take a similar approach. They are structured so that students are introduced to concepts, principles, and procedures in their text and through lectures, which they are then asked to apply to many problem-solving exercises. The hope is that, with extensive practice, students will integrate [[procedural]] and [[conceptual knowledge]] and develop expert-like schemata and planning skills. Unfortunately, many students don’t. In addition to exiting these courses with lingering naïve misconceptions (e.g., Halloun & Hestenes, 1985; McDermott, 1984), they continue to solve problems largely by manipulating equations until the desired quantity is isolated—that is, by means-end analysis—instead of by identifying relevant principles and generating solution plans before starting to solve a problem, as experts do (Dufresne et al., 1992; Larkin, McDermott, Simon, & Simon, 1980; Priest & Lindsay, 1992).

We refer to the traditional approach to physics instruction described in the preceding paragraph as [[implicit instruction]], because it encourages the inductive development of abstractions (concepts, principles, and schemata) through repeated exposure to instances, instead of by explicitly reifying these abstractions (O'Malley & Chamot, 1994). In response to the limitations of implicit approaches to physics instruction that neither ask students to plan nor scaffold them in doing it, several instructional scientists have proposed methods that explicitly engage students in planning exercises (Dufresne et al., 1992; Leonard, Dufresne, & Mestre, 1996; Mestre, Dufresne, Gerace, & Hardiman, 1993). These methods have met with modest success when tested mainly using high-achieving students (B or above in an introductory college physics course). We suggest that the main reason that many students, even high achievers, are unable to use explicit planning methods effectively is that they lack the basic concepts, principles, and procedures that are prerequisites for effective planning.

This project tested the effectiveness of engaging students in reflective dialogues after they solve problems in Andes (e.g., Gertner & VanLehn, 2000) which explicitly targeted the three main types of knowledge that experts employ during planning: knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). Explicit instruction on these knowledge components proved to be effective and more efficient than giving students repeated practice in solving different types of problems.

=== Dependent Variables ===
* ''Gains in qualitative and quantitative knowledge''. Post-test score,and pre-test to post-test gain scores, on near and far [[transfer]] items.
* ''Short-term retention''. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
* ''[[Long-term retention]]''. Performance on final exam, taken several weeks after the intervention.

=== Independent Variables ===

Students within each course section were block-randomly assigned to one of two dialogue conditions: standard [[Knowledge Construction Dialogues]] (KCDs) or control with no KCDs. Students in the control condition were assigned an additional five Andes problems to attempt to better equate time on task.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

* Number of problems completed before the post-test was administered
* Number of dialogues that the student completed
* Grade point average (CQPR)
* College major
* Pre-test score

=== Hypothesis ===
The main hypothesis tested was that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems.

=== Findings ===
Analyses of pre- and post-test scores were more encouraging than for last year's study ([[Post-practice reflection (Katz)]]). After omitting scores from one student with a post-test duration of less than two minutes, we were left with treatment and control groups of equal size (''n'' = 33). We also re-classified two treatment subjects who did no dialogues as control subjects (effective treatment and control ''n''s = 31 and 35, respectively), although these subjects were not able to access the five extra control-group Andes problems. As we had hoped, ANOVA showed no significant differences between the effective treatment and control groups on pre-test score (respective ''M''s = 12.10 and 11.77; ''F'' < 1), but treatment subjects had higher mean post-test scores (17.97 vs. 15.57; ''F''(1, 64) = 4.89, ''p'' = .031), mean raw gain scores (5.87 vs. 3.80; ''F'' = 5.62, ''p'' = .021), and mean Estes gain scores (0.330 vs. 0.208; ''F'' = 6.74, ''p'' = .012) than did control subjects. In short, without regard to problem-solving practice, subjects who did KCDs did significantly better on the post-test than those who did not.

Student participation in both homework problem solving and dialogues was much improved over last year, with 21 of 34 control subjects (62%) and 23 of 33 treatment subjects (70%) finishing at least 80% of the assigned target problems (of 31 for control, 26 for treatment) prior to the post-test and with 25 treatment subjects (76%) finishing at least 80% of the 26 associated KCDs. However, participation was
still far from perfect; 7 treatment subjects (21%) completed half or fewer of the assigned KCDs on time, and 2 of them completed no target problems or KCDs on time. We again treated KCD completion and target problem completion as continuous independent variables in regression analyses. Regressing post-test score on pre-test score, QPA, number of KCDs completed, and number of target problems completed
(''R''2 = .52, ''F''(4, 61) = 16.70, ''p'' < .00001) showed positive contributions of all factors, but only pre-test score, QPA, and KCD completion were statistically significant (''p''s < .001, .05, & .05, respectively); problem completion was ''ns'' (''p'' = .54). Therefore, across all subjects it was KCD completion, as opposed to target homework problem completion, that significantly predicted post-test performance.

To measure retention and transfer, we also analyzed scores from an hourly exam administered by one instructor to his two sections (''n''=47). All problems on this exam were quantitative and covered a subset of the units targeted during the intervention. Scores on this exam ranged from 176 to 382 (of 400), and were highly correlated with
pre- and post-test scores; ''r''s(45) = .54 and .71, ''p''s < .0001 and .00001. However,
ANOVAs showed no differences between groups (Fs < 1), and regression of subscores on QPA, KCDs completed, and target problems completed (''R''2 = .54, ''F''(3, 43) = 16.58,
''p'' < .00001) showed positive contributions of all factors but a significant effect of only QPA (''p'' < .00001). Therefore, student performance on the hourly exam was not significantly affected by either KCD completion or target problem completion.

In summary, student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed prior to the post-test. However, neither measure had a significant effect on student performance on an exam that focused on
problem-solving ability.

=== Explanation ===

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s central hypothesis is that robust learning occurs when two conditions are met:

* ''The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work''. Since this experiment did not deal with collaboration between agents, it did not test this condition. It did, however, show that the more that students engage in one form of learning-by-doing—mainly, post-practice reflection—the more they learn.
* ''The student takes the learning-by-doing path unless it becomes too difficult''. More students took the learning-by doing path than in our prior study, perhaps because this year instructors ''required'' them to do so (i.e., this time there were negative consequences for avoiding them).

Analyses of retention and transfer to problem-solving ability did not reveal a significant effect of the reflective dialogues. We had hoped that the capstone dialogues that treatment students completed before selected problems, at the end of each unit, would support problem-solving ability. It is possible that the intervention consisted of too few capstone dialogues (only five) to observe any effect.

What seems to be the critical feature of post-practice reflection (PPR) in supporting conceptual understanding is the explicit instruction that it provides in domain knowledge. PPR may help students to fill in knowledge gaps, resolve misconceptions,
and abstract from the case at hand so that they are better prepared to engage in constructive activity (e.g., self-explanation) in future problems. Preliminary research comparing Andes (which encourages implicit learning of problem-solving strategies) with Pyrenees, a system that teaches problem-solving strategies explicitly, also
suggests an advantage for explicit instruction (VanLehn et al., 2004). Further research is needed to identify the mechanisms that drive learning from reflective dialogue, and to increase its potential to enhance problem-solving ability in addition to conceptual knowledge.

=== Further Information ===
==== Annotated bibliography ====
* Presentation to site visitors, 2005
* Full paper accepted at AIED 2007:
**Katz, S., Connelly, J., & Wilson, C. (in press). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. ''Proceedings of AIED07''.

==== References ====
*Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), ''Advances in the Psychology of Human Intelligence, Vol. 1'' (pp. 7-75). Hillsdale, NJ: Erlbaum.
*Dufresne, R. J., Gerace, P., Hardiman, T., & Mestre, J. P. (1992). Constraining novices to perform expertlike analyses: Effects on schema acquisition. ''The Journal of the Learning Sciences, 2'' (3), 307-331.
*Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), ''ITS 2000: Proceedings of the 5th International Conference on Intelligent Tutoring Systems'' (pp. 133-142). Berlin: Springer-Verlag.
*Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. ''Science, 208'', 1335-1342.
*Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. ''American Journal of Physics, 64'' (12), 1495-1503.
*Mestre, J. P., Dufresne, R. J., Gerace, W. J., & Hardiman, P. T. (1993). Promoting skilled problem-solving behavior among beginning physics students. ''Journal of Research in Science Teaching, 30'', 303-317.
*O’Malley, M., and Chamot, A. (1994). ''The CALLA Handbook''. Reading, MA: Addison-Wesley.
*Priest, A. G., & Lindsay, R. O. (1992). New light on novice-expert differences in physics problem solving. ''British Journal of Psychology, 83'', 389-405.
*VanLehn, K., Bhembe, D., Chi, M., Lynch, C., Schulze, K., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2004). Implicit versus explicit learning of strategies in a nonprocedural cognitive skill. In J. C. Lester, R. M. Vicari, & F. Paraguacu, (Eds.), ''Intelligent Tutoring Systems: 7th International Conference'' (pp. 521-530). Berlin: Springer-Verlag.

==== Connections ====
This project shares features with the following research projects:

Use of Questions during learning
* [[Post-practice reflection (Katz)]]
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[FrenchCulture | FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)]]
* [[Rummel_Scripted_Collaborative_Problem_Solving|Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)]]

Self explanations during learning
* [[Craig_questions|Deep-level questions during example studying (Craig & Chi)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & Chi)]]
* [[Hausmann Study | A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)]]

==== Future plans ====
Our future plans for January 2008 - December 2008:
* write journal article expanding AIED conference paper

[[Category:Study]]

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-10T21:05:11Z

Connelly: /* Abstract */

Extending Reflective Dialogue Support (Katz & Connelly)

2008-04-10T20:54:46Z

Connelly: /* Abstract */