Conceptual Learning in Chemistry

Supporting Conceptual Learning in Chemistry through Collaboration Scripts and Adaptive, Online Support

Bruce M. McLaren, Nikol Rummel, Andreas Harrer, Hans Spada, Niels Pinkwart

Overview

PI: Bruce M. McLaren

Co-PIs: Nikol Rummel, Andreas Harrer, Hans Spada, Niels Pinkwart

Others who have contributed 160 hours or more:

• Dimitra Tsovaltzi, University of Saarland, Germany, experimental design and execution
• Isabel Braun, Freiburg University, Germany, experimental design and execution
• Oliver Scheuer, University of Saarland, Germany, data mining and programming
• Roger Miller, University of Saarland, Germany, programming

Abstract

Chemistry students, like students in physics, mathematics, and other technical disciplines, often learn to solve problems algorithmically, applying well-practiced procedures to textbook problems. But often these students do not understand the underlying conceptual aspects of the problems they solve algorithmically. An important setting for promoting conceptual understanding in chemistry is the laboratory, where students must apply not only pre-defined problem solving procedures, but must also plan experiments, hypothesize outcomes, conduct and monitor experiments, and evaluate outcomes. In the PSLC Chemistry LearnLab, the Virtual Laboratory (VLab) is the online software environment used to simulate a real chemistry laboratory and assist students in their conceptual understanding of chemistry. However, the VLab on its own is not enough. We propose to further assist chemistry students in their conceptual learning, first, through having pairs of students collaborate on problems, assisted by computer-mediated collaboration scripts that extend the VLab and, later, through dynamic adaptation of those collaboration scripts. In this one-year project, we will analyze current use of the VLab, design and implement a computer-mediated collaborative environment around the VLab, using a collaborative software tool called Cool Modes, and execute a lab study to evaluate the effectiveness of this tool. The one-year PSLC project will provide the foundation for an externally-funded project, still conducted within the PSLC Chemistry LearnLab, in which we will perform full-scale in vivo studies to test the hypotheses that (1) collaboration, supported by collaboration scripts, can promote the creation and strengthening of conceptual chemistry knowledge and (2) that dynamic adaptation of the collaboration scripts can further improve that learning.

Glossary

See Scripted Collaborative Problem Solving Glossary

Research Questions

Does collaboration – and in particular scripted collaboration – improve students’ robust learning, and in particular conceptual learning, in the domain of chemistry?

Does the script approach improve students’ collaboration, and does this result in more robust learning of the chemistry content?

Hypothesis

These research questions led us to the following two hypotheses:

H1

Computer-mediated collaboration, facilitated by collaboration scripts and added to experimental exercises within the stoichiometry course, can promote the creation and strengthening of conceptual stoichiometry knowledge components.

H2

Computer-mediated collaboration, facilitated by adaptive collaboration scripts and added to experimental exercises within the stoichiometry course, can promote the creation and strengthening of conceptual stoichiometry knowledge components.

Background and Significance

A central issue in chemistry education is teaching students to problem solve conceptually rather than simply apply mathematical equations. Research in chemistry education has shown that students tend to learn and solve problems “algorithmically” but often do not grasp the deeper conceptual aspects of chemistry and reasoning necessary to be more creative and flexible problem solvers (Gabel & Bunce, 1994; Bodner & Herron, 2002). Dave Yaron, the chair of the Pittsburgh Science of Learning Center (PSLC)’s Chemistry LearnLab, has expressed a similar view in observing his students, saying, “many students learn the mathematical tools necessary to solve chemistry problems but don’t know when to appropriately apply those tools” (Personal communication between McLaren and Yaron February 27, 2006; also discussed in Yaron et al, 2003). The phenomenon that learners have problems transferring instructed procedures to new problems due to a lack of conceptual understanding has been observed and investigated also in other domains, for example, math (Singley & Anderson, 1989). The difficulty chemistry students have can be viewed as a transfer problem, an important area of investigation in the PSLC’s emerging theory of robust learning. In particular, while chemistry students often have success on problems that are very similar to ones illustrated in a textbook or demonstrated in a classroom, they tend to struggle with problems that could be solved with similar techniques but are not obviously of the same type (e.g., the source and target problems don’t share surface features). This difficulty is due to students lacking the conceptual understanding of chemistry to recognize similar core problems that come in “different clothes.” There is some descriptive evidence in chemistry education research indicating that collaborative activities can improve conceptual learning in chemistry (e.g. Towns & Grant, 1998; Fasching & Erickson, 1985). Other studies, while not focused specifically on conceptual versus algorithmic learning, have demonstrated increased performance as well as motivational benefits of collaborative learning in chemistry (Ross & Fulton, 1994). On the other hand, none of these collaborative learning studies in chemistry was a randomized controlled experiment. In general, there is a lack of controlled experimentation on the potential benefits of collaborative learning in chemistry. However, such evidence exists in math (Berg, 1993, 1994), physics (Hausmann, Chi & Roy, 2004; Ploetzner, Fehse, Kneser, & Spada, 1999), or scientific experimentation (Teasley, 1995). Research in collaborative learning has shown promise in helping students to more deeply process information and thus improve their conceptual learning. A few different mechanisms are accountable for the benefits of collaborative activities, like giving explanations to the partner, receiving help from the partner after making a mistake or asking for help, and co-constructing or jointly negotiating knowledge (Hausmann, Chi, & Roy, 2004; Ploetzner, Dillenbourg, Preier, & Traum, 1999; Webb, 1989; Webb, Trooper, & Fall, 1995). In sum, results from this research lead us to the assumption that it would be worthwhile investigating the advantages of collaborative activities on the acquisition of robust, transferable conceptual knowledge in controlled experimental studies in chemistry. In the proposed project, we will test the hypothesis that a computer-supported collaborative learning system can help students improve their conceptual understanding of chemistry. Our goal is to help students actively process the material they encounter, moving them away from the mechanical, algorithmic approach taken by many chemistry students. In terms of the PSLC theoretical framework, we assume that the collaborative situation creates additional learning events through the above cited mechanisms of receiving help, giving explanations, and co-constructing knowledge. In addition, the collaborative setting may increase the likelihood that students capitalize on the learning events offered by the domain setting (i.e. the chemistry learning environment). That is, collaborative interactions can increase the likelihood of particular path choices in the learning event space that benefit learning. To test our hypothesis, we will develop collaborative extensions to Dave Yaron’s online stoichiometry course, an established part of the PSLC Chemistry LearnLab, and compare individual learning in the course with scaffolded collaborative learning. Before we devise specific support features for the existing online, stoichiometry course, however, our intention is to analyze extant student data from the course to determine the aspects of the current educational materials that could most benefit from collaborative learning. Nevertheless, we have preliminary ideas about the way we will support collaboration as part of the course, based on some of our past research. In particular, we plan to support collaborating students through the use of collaboration scripts, prompts, questions, and assigned roles that guide students through collaborative work (e.g., Kollar, Fischer & Hesse, 2003; O’Donnell, 1999). Much research has shown that fruitful collaboration does not generally occur by itself (Dillenbourg, Baker, Blaye, & O’Malley, 1995; Rummel & Spada, 2005a). Collaborative partners often do not engage in productive interactions and thus miss the opportunity to benefit from their collaboration. In order to ensure that students can actually profit from their collaboration, it is important that collaborative partners learn how to work together in productive ways. Particularly at the beginning, guidance, instruction, and training are required to achieve effective collaboration (Slavin, 1996). Research in the area of collaborative inquiry learning, particularly relevant to the experimental framework of Yaron’s online stoichiometry course, has also uncovered a need for scaffolded collaboration (Bell, Slotta, Schanze, to appear). Moreover, we believe that it might be best to scaffold collaboration in an adaptive fashion, emphasizing and fading structured support for collaboration according to the particular needs of the specific collaborators. For instance, some work has uncovered the dangers of over-scripting, i.e., providing too much structure and support for collaboration (Dillenbourg, 2002). Identifying and being sensitive to such situations in real time will require adaptation. Some of our own work suggests this direction, too: Results of one study (Rummel & Spada, 2005b) indicated that collaboration scripts were beneficial both to collaboration and domain learning. However, in a more recent study Rummel, Spada, & Hauser (in press) found that students observing a model of collaboration (i.e., a worked collaboration example) collaborated better and learned more than students who followed a script. Rummel and colleagues concluded that one problem with scripting may have been that students were overwhelmed by the concurrent demands of collaborating, following the detailed script instructions, and trying to reflect on the scripting on a meta-level in order to learn. Taken together, these studies suggest that different students, under different circumstances, may benefit from different types of collaboration support; thus, a collaborative learning system that can adapt its support might prove quite powerful. One study that we are aware of, in which adaptive, strategic “prompts” in a collaborative system were shown to lead to productive collaboration and support for learning, is the work of Gweon, Rosé, Carey, & Zaiss (2006). While interest in adaptive collaborative learning systems is on the rise in the computer-supported collaborative learning (CSCL) community (Soller, Jermann, Muehlenbrock, & Martinez, 2005), little progress has yet been made on the implementation of adaptive support. Further evidence for our assumption that adaptive support of collaboration will be most effective for learning also comes from research on cognitive tutors (e.g. Anderson, Corbett, Koedinger, & Pelletier, 1995). A key strength of cognitive tutors is that they provide just-in-time support, tailored to the needs of the individual student in a particular moment. As soon as the student makes an error, he or she receives feedback from the system and usually is given some advice about how to overcome the impasse. In the long run, this is what our adaptive collaboration approach aims at: a collaboration tutor. However, this is obviously an overly ambitious goal to achieve within the initial one-year project. Collaboration as a domain is particularly challenging, because it is very difficult – perhaps impossible – to define all possible paths in the collaboration space ahead of time or to define a “best path” or “buggy paths” through the space. It is only possible to define positive and negative collaborative behaviors in general terms. The challenge is to find ways to monitor those behaviors based on real-time data collected during student collaboration and have the system respond to the collaborating partners accordingly. In order to make it more tractable for a computer system to respond to a student, yet also supply helpful collaboration support, one promising idea we intend to explore is setting up the collaboration such that only particular actions, and in turn particular paths through the collaboration space, can lead to correct outcomes. In this way, we can make inferences about the collaboration from the observable actions, rather than from natural language content. For instance, suppose that two students are collaborating on a chemistry problem in which they are asked to perform an experiment and answer questions about a particular chemical reaction. Suppose further that one student has some of the available resources (i.e., substances, beakers, meters, etc.) and the other student has other (different) resources necessary to solve the problem. A “positive” solution path for this problem is one in which the students combine substances of equal amounts in a beaker. If the students take this action, then one could assume that either (a) they both understood the concept of dilution underlying this step or (b) one student explained the concept to the other. In other words, their collaboration led to what appears to be a successful conceptual understanding, even though we can’t know precisely how this happened. On the other hand, if they combine substances of unequal quantity, this action may indicate a “buggy path” and an opportunity to scaffold the students’ interaction (i.e., “Can you explain why you combined substances of unequal quantity?”). While it could be argued that a collaboration setup such as this is “artificial” – in fact Micki Chi made an argument against such “artificial” collaboration setups at a recent PSLC luncheon (June 12, 2006) – we would argue that the goal is not to provide a natural collaborative setting during the experimental intervention. Rather, our goal is to provide a framework and scaffolds for collaboration that can ultimately be faded, once the students demonstrate good collaborative behavior and learning. Our hypothesis is that such structure will help students learn productive collaborative behaviors, ones that lead to content learning, and that potentially allow them to subsequently collaborate successfully without support. In this project plan we request limited seed funding from the PSLC for only the first year of what we project to be a four-year project. Our goal in the first year will be to analyze problem solving in the stoichiomery course, both individual and collaborative, and perform small-scale (i.e., small N) lab studies to experiment with the general hypothesis that collaboration support, in the form of collaboration scripts, can enhance conceptual learning of stoichiometry. The first year will also involve technical implementation, in particular a prototype development of an online collaboration system that integrates the VLab, an online simulation of chemistry experimentation and an integral part of the PSLC Chemistry LearnLab (Yaron et al, 2003), with Cool Modes, a software tool that facilitates computer-mediated collaboration and simulation (Pinkwart, 2003). An additional technical development during the first year, but funded separately by an Office of Naval Research grant (PIs Koedinger, Aleven, and McLaren), will be work on bootstrapping, a technique for collecting and dynamically evaluating collaborative (or single student) behavior, continuing previous work by McLaren and Harrer (Harrer et al, in press; Harrer, McLaren, et al, 2005; McLaren et al, 2005; McLaren et al, 2004). As the bootstrapping technique is developed on the ONR project, the stoichiometry data collected on this project will be run against and tested using bootstrapping. This technical work will then set the stage for the adaptive collaboration effort we propose in the second phase of the project. The second phase of the project, after the initial year of PSLC seed funding, will focus on full-scale in vivo experimentation, first testing collaboration scripts and, later, adaptive collaboration scripts. Toward the end of the first year we will submit a proposal to the DFG, the German equivalent of NSF. If successful, we will continue the work over a subsequent three-year period using the PSLC infrastructure and Chemistry LearnLab as a means to support and perform in vivo experiments. Thus, while the first year of the project, focused on setting the groundwork of the project, will not involve in vivo experimentation, the longer-term emphasis will be on such experimentation. The second phase of the project will also involve continued technical development to support dynamic adaptation of collaborative support, continuing work on the bootstrapping technique, described above, and investigating the use of cognitive tutoring techniques to support collaboration.

Independent Variables

To test our hypotheses and the effect of personalization and worked examples on learning, we designed and have executed two 2 x 2 factorial studies.

• One independent variable is Personalization, with one level impersonal instruction, feedback, and hints and the other personal instruction, feedback, and hints.
• The other independent variable is Worked Examples, with one level tutored problem solving alone and the other tutored problem solving together with worked examples. In the former condition, subjects only solve problems using the intelligent tutor; no worked examples are presented. In the latter condition, subjects alternate between observation and self-explanation of a worked example and solving of a tutored problem. This alternating technique has yielded better learning results in prior research (Trafton and Reiser, 1993).

With respect to personalized language (and because we got a null result in the first two studies), we thought that perhaps our conceptualization and implementation might not be as socially engaging as we had hoped. This was also suggested to us by Rich Mayer, who reviewed the first study. In a recent study that Mayer and colleagues did (Wang, Johnson, Mayer, Rizzo, Shaw, & Collins, in press), based on the work of Brown and Levinson (1987), they found that a polite version of a tutor, which provided polite feedback such as, “You could press the ENTER key”, led to significantly better learning than a direct version of the tutor that used more imperative feedback such as, “Press the ENTER key.” We decided to investigate this in a third in vivo study in which we changed all of the personalized instruction, feedback, and hints of the tutor to more polite forms, similar to that used by Mayer and colleagues.

Thus, for the third study we changed the first independent variable to "Politness," with one level polite instruction, feedback, and hints and the other direct instruction, feedback, and hints. The 2 x 2 factorial design for our third and most recent study is shown below.

Below is a table that provides examples of the differences in language between the polite version of our tutor and earlier versions.

Dependent Variables

To evaluate learning, students are asked to solve pre and post-test stoichiometry problems that are isomorphic to one another and to the tutored problems. Thus, we have focused on normal post-tests in our studies so far. When (and if) we see an effect in a normal post-test, we will conduct studies to test retention.

Findings

In two initial 2 x 2 factorial studies, we found that personalized language and worked examples had no significant effects on learning, thus not supporting hypotheses H1 and H2. On the other hand, there was a significant difference between the pre and posttest in all conditions, suggesting that the intelligent tutor present in all conditions did make a difference in learning. For study 1 we had N = 63 and for study 2 we had N = 76. The results of Study 1 are reported in (McLaren, Lim, Gagnon, Yaron, and Koedinger, 2006). We are currently analyzing the data to test hypothesis H3, that is, to determine if learning with worked examples was more efficient.

One possible explanation for why neither personalized language nor worked examples have made a difference thus far is the switch from a lab environment to in vivo experimentation. Most of the results from past studies of both personalized language and worked examples come from lab studies, so it may simply be that the realism and messiness of an in vivo study makes it much more difficult for interventions such as these to make a difference to students’ learning. It may also be that the tutoring received by the subjects simply had much more effect on learning than the worked examples or personalized language.

We recently concluded the third study in which we investigated the use of polite language, rather than personalized language (as shown in the table above). We have so far only analyzed the first 33 subjects, out of N=84 (for details on the analysis of the first 33 subjects see McLaren, Lim, Yaron, & Yaron, in press). The preliminary data indicates that the polite condition leads to larger learning gains than the non-polite condition, however, not at a statistically significant level. Worked examples also did not make a difference to learning. Thus, once again, hypotheses H1 and H2 were not supported. We are in the process of analyzing the rest of the data from the remaining subjects who participated in this study, including an investigation of hypothesis H3 and the efficiency of learning.

Explanation

This study is part of the Coordinative Learning cluster. The study follows the Coordinative Learning hypothesis that two (or more) sources of instructional information can lead to improved robust learning. In particular, the study tests whether an ITS and personalized (or polite) language used together lead to more robust learning and whether an ITS and worked examples used together lead to more robust learning.

Connections to Other PSLC Studies

• The key finding of our studies so far has been that learning has not improved when students use an intelligent tutor in conjunction with other instructional techniques. Two other studies in the Coordinative Learning cluster, the Aleven/Butcher and Booth/Siegler/Koedinger/Rittle-Johnson projects, and one study in the Interactive Communication cluster, the Renkl/Aleven/Salden project, are also investigating the complementary effects of intelligent tutoring combined with another instructional technique.

• Our project differs from the Booth study in that we are using only correct examples as a means to strengthen correct knowledge components. We are not using incorrect examples to weaken incorrect knowledge components, as Booth is testing in her study.

• Our project also relates to the Aleven and Butcher project in that we both are exploring the learning value of e-Learning Principles (they are investigating Contiguity; we are looking at personalization and worked examples). In addition, like their study, we are prompting self-explanation as a means to promote robust learning.

Annotated Bibliography

• McLaren, B. M., Lim, S., Yaron, D., and Koedinger, K. R. (2007). Can a Polite Intelligent Tutoring System Lead to Improved Learning Outside of the Lab? In the Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED-07), pp 331-338. [pdf file]
• McLaren, B. M., Lim, S., Gagnon, F., Yaron, D., and Koedinger, K. R. (2006). Studying the Effects of Personalized Language and Worked Examples in the Context of a Web-Based Intelligent Tutor; In the Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS-2006), pp. 318-328. [pdf file]
• McLaren, B. M. Presentation to the NSF Site Visitors, June, 2006.

References

• Aleven, V. & Ashley, K. D. (1997). Teaching Case-Based Argumentation Through a Model and Examples: Empirical Evaluation of an Intelligent Learning Environment, Proceedings of AIED-97, 87-94.
• Beck, I., McKeown, M. G., Sandora, C., Kucan, L., and Worthy, J. (1996). Questioning the author: A year long classroom implementation to engage students in text. Elementary School Journal, 96, 385-414.
• Brown, P. and Levinson, S. C. (1987). Politeness: Some Universals in Language Use. Cambridge University Press, New York.
• Clark, R. C. and Mayer, R. E. (2003). e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer.
• Conati, C. and VanLehn, K. (2000). Toward Computer-Based Support of Meta-Cognitive Skills: a Computational Framework to Coach Self-Explanation. Int’l Journal of Artificial Intelligence in Education, 11, 398-415.
• Gott, S. P., Lesgold, A., & Kane, R. S. (1996). Tutoring for Transfer of Technical Competence. In B. G. Wilson (Ed.), Constructivist Learning Environments, 33-48, Englewood Cliffs, NJ: Educational Technology Publications.
• Kolb, D. A. (1984). Experiential Learning - Experience as the Source of Learning and Development, Prentice-Hall, New Jersey. 1984.
• Mathan, S. and Koedinger, K. R. (2002). An Empirical Assessment of Comprehension Fostering Features in an Intelligent Tutoring System. Proceedings of ITS-2002. Lecture Notes in Computer Science, Vol. 2363, 330-343. Berlin: Springer-Verlag
• Mathan, S. (2003). Recasting the Feedback Debate: Benefits of Tutoring Error Detection and Correction Skills. Ph.D. Dissertation, Carnegie Mellon Univ., Pitts., PA.
• Mayer, R. E., Johnson, W. L., Shaw, E. and Sandhu, S. (2006). Constructing Computer-Based Tutors that are Socially Sensitive: Politeness in Educational Software, International Journal of Human-Computer Studies 64 (2006) 36-42.
• Moreno, R. and Mayer, R. E. (2000). Engaging students in active learning: The case for personalized multimedia messages. Journal of Ed. Psych., 93, 724-733.
• Paas, F. G. W. C. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive load approach. Journal of Ed. Psych., 84, 429-434.
• Renkl, A. (1997). Learning from Worked-Out Examples: A Study on Individual Differences. Cognitive Science, 21, 1-29.
• Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295-312.
• Trafton, J. G. and Reiser, B. J. (1993). The contributions of studying examples and solving problems to skill acquisition. In M. Polson (Ed.) Proceedings of the 15th annual conference of the Cognitive Science Society, 1017-1022.
• Wang, N., Johnson, W. L., Mayer, R. E., Rizzo, P., Shaw, E., & Collins, H. (in press). The Politeness Effect: Pedagogical Agents and Learning Outcomes. To be published in the International Journal of Human-Computer Studies.