Rummel Scripted Collaborative Problem Solving

Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving

Nikol Rummel, Dejana Diziol, Bruce McLaren, and Hans Spada

Abstract

In this project, the Algebra I Cognitive Tutor is extended to a collaborative learning environment: students learn to solve system of equations problems while working in dyads. As research has shown, collaborative problem solving and learning has the potential to increase elaboration on the learning content. However, students are not always able to effectively meet the challenges of a collaborative setting. To ensure that students capitalize on collaborative problem solving with the Tutor, a collaboration script was developed that guides their interaction and prompts fruitful collaboration. During the scripted problem solving, students alternate between individual and collaborative phases. Furthermore, they reflect on the quality of their interaction during a recapitulation phase that follows each problem they solve.

To assess the effectiveness of the script, we conducted two classroom studies. In an initial, small scale study (pre study) that served to establish basic effects and to test the procedure in a classroom setting, we compared scripted collaboration with an unscripted collaboration condition in which students collaborated without support. In the full study, we furthermore compared these collaborative conditions to an individual condition to assess the effect of the collaborative Tutor extension to regular Tutor use. The experimental learning phase took place on two days of instruction. On the third day, during the test phase, students solved several post tests that assessed robust learning. Analyses of the collected measures are in progress.

Background and Significance

In our project, we combined two different instructional methods both of which have been shown to improve students’ learning in mathematics: Learning with intelligent tutoring systems (Koedinger, Anderson, Hadley, & Mark, 1997) and collaborative problem solving (Berg, 1993). The Cognitive Tutor Algebra that was used in our study is a tutor for mathematics instruction at the high school level. Its main features are immediate error feedback, the possibility to ask for a hint when encountering impasses, and knowledge tracing, i.e. the Tutor creates and updates a model of the student’s knowledge and selects new problems tailored to the student’s knowledge level. Although several studies have proven its effectiveness, students do not always benefit from learning with the Tutor. First, because the Tutor places emphasis on learning problem solving skills, yet a deep understanding of underlying mathematical concepts is not necessarily achieved (Anderson, Corbett, Koedinger, & Pelletier, 1995). Second, students do not always make good use of the learning opportunities provided by the Cognitive Tutor. So far, the Cognitive Tutor has been used in an individual learning setting only. However, as research on collaborative learning has shown, collaboration can yield elaboration of learning content (Teasley, 1995), thus this could be a promising approach to reduce the Tutor’s shortcomings. On the other hand, students are not always able to effectively meet the challenges of a collaborative setting (Rummel & Spada, 2005). Collaboration scripts have proven effective in helping people meet the challenges encountered when learning or working collaboratively (Kollar, Fischer, & Hesse, in press). To ensure that students would profit from a collaboratively enhanced Tutor environment, we thus developed a collaboration script that prompts fruitful interaction on the Tutor while students learned to solve “systems of equations”, a content novel to the participating students. The script served to goals: First, to improve students’ robust learning of knowledge components relevant for the systems of equation units (i.e. both problem solving skills and understanding of the concepts). Second, students should increase their explicit knowledge of what constitutes a fruitful collaboration and improve their collaborative skills in subsequent interactions without further script support. This is particularly important due to the risk of overscripting collaboration that has been discussed in conjunction with scripting for longer periods of time (Rummel & Spada, 2005).

Glossary

• Collaboration scripts: Collaboration scripts structure the collaboration process by guiding the interacting partners through a sequence of interaction phases with designated activities and roles. Scripts are expected to promote learning by prompting cognitive, meta-cognitive and social processes that might otherwise not occur, i.e. students are more likely to traverse useful learning paths than in unscripted collaboration. For example, the script prompts interacting partners to engage in activities like posing questions, providing explanations, and giving feedback.

• Collaborative Problem-Solving Script (CPS): The CPS is the script approach that is implemented in the experimental condition of the study. For each problem the dyad solves on the Cognitive Tutor, their interaction is structured in an individual problem-solving phase, a collaborative problem-solving phase, and a recapitulation phase. The script further provides guidance during the collaborative problem-solving phase.

• Individual Problem-Solving Phase: Both students solve a pre-task on their own as a preparation for the complex story problem they solve together during the following collaborative problem-solving phase.

• Collaborative Problem-Solving Phase: Students’ interaction is structured in several problem solving steps. For every step, students receive instructions prompting them to engage in collaborative behaviors that have been shown to increase learning. Furthermore, students receive adaptive instructional support when meeting impasses during the problem-solving process.

• Recapitulation Phase: Students engage in a reflection of the group process: They evaluated (i.e. rate) their collaboration on eight dimensions and set goals for how to improve it during the next joint problem-solving.

Research question

Does collaboration – and in particular scripted collaboration – improve students’ robust learning in the domain of algebra?

Does the script approach improve students’ collaboration, and does this result in more robust learning of the algebra content?

Independent variables

The independent variable was the mode of instruction. Three conditions were implemented in the learning phase that took place on two days of instruction:

• Individual condition: Students solved problems with the Algebra I Cognitive Tutor in the regular fashion (i.e. computer program as additional agent).
• Unscripted collaborative condition: Students solved problems on the Cognitive Tutor while working in dyads; thus, a second learning resource was added to the regular Tutor environment (i.e., both a peer and a computer program as additional agents).
• Scripted collaboration condition: As in the unscripted collaborative condition, students solved problems on the Tutor while working in dyads; however, their dyadic problem solving was guided by the collaborative problem-solving script (i.e., both a peer and a computer program as additional agents).

The Cognitive Tutor supported students’ problem solving by flagging their errors and by providing on-demand hints.

Communication between the collaborative partners in the collaborative conditions was face to face: they joined together at a single computer to solve the problems. In the scripted collaboration condition, the script structured the allocation of subtasks between the two students.

In the pre study that aimed at establishing basic effects and at testing the procedure in a classroom setting, we only compared unscripted and scripted collaborative problem solving.

Hypothesis

The hypothesis can be stated in two parts: knowledge gains in the domain of algebra and improvement of the dyads’ collaboration skills:

1. We expect the collaborative conditions – in particular the scripted collaboration condition – to outperform the individual conditions in measures of robust learning of the algebra content.
2. We expect that problem-solving with the Collaborative Problem-Solving Script will lead to a more effective interaction than unscripted collaboration in the learning phase. In addition, we expect that the newly gained collaborative skills will sustain during subsequent collaborations in the test phase when script support is no longer available.

Dependent variables

Students solved several post tests during the test phase that took place two days after instruction:

• Near transfer, immediate: Student progress on training problems during the learning phase was analyzed. Their improvement in solving system of equations, i.e. their ability to transfer from one problem to an isomorphic problem with different content, can be assessed by looking at the learning curves.
• Near transfer, retention: Two post-tests measured near transfer. As during the learning phase, students solved system of equations problems on the Cognitive Tutor. The problems were “isomorphic” to those in the instruction, but had a different content. One near transfer test was solved collaboratively to measure both students’ algebra skills and their collaboration skills, the other near transfer test was solved individually to assess if the students were able to transfer the gained knowledge to isomorphic problems when working on their own.
• Far transfer: Main concepts students were supposed to learn during instruction were slope, y-intercept and intersection point. The students’ understanding of these mathematical concepts was assessed in a far transfer paper and pencil test. The items of this test tapped the same knowledge components as the problems in instruction; however, the problems where non-isomorphic to those in the instruction, thus demanded students to flexibly apply their knowledge to problems with a new format.
• Accelerated future learning: To measure accelerated future learning, students learned how to solve problems in a new area of algebra – inequality problems – on the Cognitive Tutor. According to their condition during the experimental learning phase, they either worked individually or collaboratively on this post test. However, in contrast to the learning phase, they did not receive additional instructional support.

Several variables from these post tests are compared to test the two hypothesis.

1. For the post tests conducted on the Cognitive Tutor (retention and accelerated future learning), variables such as number of errors per problem, time per problem, decrease of error rates over the course of several problems etc are extracted from the log files to compare conditions.
   For the paper and pencil post test (far transfer), we developed a scoring system. Four variables are compared:both for a problem set to basic concepts (y-intercept and slope) and for a problem set to the novel system’s concept (intersection point), we summed answers to questions with distinct answer format and open format questions.
2. To compare collaboration skills of students, we recorded students’ dialogues both during learning phase and during test phase. Based on a rating scheme from one of our former projects (Meier, Spada, & Rummel, in press), we develop a rating scheme to assess students’ interaction on skill dimensions as “giving elaborated explanations to the partner”.
   Furthermore, the paper and pencil post test entailed an open format question asking students to explain what is important for a fruitful collaboration. The answers to this question are also coded and compared between conditions

Method

The study procedure took place during 3 classroom periods over the course of a week. During day 1 and day 2 (the learning phase), the experimental conditions (i.e. the independent variables) were implemented. On day 3, students had to solve several post-tests that assessed different aspects of robust learning.

Two in vivo classroom studies were conducted: a pre study with 3 classrooms, and a full study with 8 classrooms. The pre study aimed at establishing basic effects of the script and testing the procedure in a classroom setting, thus we only compared scripted to unscripted collaboration. In the full study, we compared the scripted collaboration condition to both an individual condition and an unscripted collaboration to evaluate the merits of both enhancing the Tutor with collaboration in general and scripted collaboration in particular. Due to the disruptiveness of students in the same class using different interventions, we used a between-class design.

Participants pre study:

The unscripted condition consisted of two classes (12 and 4 students), and the scripted condition consisted of one class (13 students). All classes were taught by the same teacher.

Participants full study:

• Individual condition: 1 class (21 students)
• Unscripted condition: 3 classes (9 + 8 + 8 dyads = 25 dyads)
• Scripted condition: 4 classes (8 + 8 + 9 + 9 dyads = 34 dyads)

Findings

Findings pre study:

First results of the paper and pencil test (far transfer) revealed substantial differences between conditions. The analysis was restricted to students who always worked collaboratively when present, as we are interested in the script’s effect on collaborative learning in particular. Due to student absenteeism, only 9 students in the unscripted and 10 students in the scripted condition were included in our analysis.

Significant differences were found both in the MANOVA (Pillai-Spur, F(4, 14) = 7.35, p < .05) and in subsequent ANOVAs. Multiple choice answers on basic concepts did not show a significant difference, F(1,17) = 2.26, ns. However, the scripted condition outperformed the unscripted condition on the discrete answer questions about the system’s concept, F(1,17) = 22.16, p < .01. A significant difference between conditions was also found for the open format questions of both problem sets with F(1,17) = 5.85, p < .05 for the basic concepts and F(1,17) = 17.01, p < .01 for the system’s concept. Particularly the results of the open format questions demonstrate the script’s effect on conceptual knowledge acquisition: after scripted interaction during the learning phase, students were better at articulating their mathematical thinking compared to their unscripted counterparts. However, it should be noted that in general students in both conditions had difficulties providing explanations, thus only reached low scores in the open format questions. The amount of wrong explanations and the number of students who did not even try to articulate their thinking was very high.

Results on tests that were administered on the Cognitive Tutor (hypothesis 1: near transfer, immediate; near transfer, retention; acceleration of future learning) as well as results of the analysis of the collaboration process (hypothesis 2) are coming soon.

Findings full study:

Coming soon.

Explanation

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. According to the IC cluster’s hypothesis, instruction that yields robust learning should be designed to have the right target paths. Furthermore, it should improve the students’ path choice, i.e., increasing the probability that students take correct paths and decreasing the probability that students take alternative paths.

By enhancing the Cognitive Tutor with collaboration, the study tried to reach these two goals in two steps:

• Individual condition: The existing Cognitive Tutor already offers correct learning paths. However, individual learning on the Cognitive Tutor has several shortcomings. For example, students don’t have the opportunity to reflect on the underlying mathematical concepts in natural language interaction in order to gain a deeper understanding. In addition, students do not always use the offered learning paths (as the opportunity to reflect on on-demand hints and error feedback) effectively.
• Unscripted collaboration condition (1st step): By enhancing the Algebra I Cognitive Tutor to be a collaborative learning environment, we added a second learning resource –the learning partner. This adds further correct learning paths to the learning space, for instance, learning by giving explanations, the possibility to request help, and learning by knowledge co-construction. However, similarly to the learning paths in an individual setting, students do not always capitalize on these learning opportunities.
• Scripted collaboration condition (2nd step): To increase the probability that students take correct learning paths, students’ interaction in the scripted collaboration condition is guided by the Collaborative Problem Solving Script. By prompting collaborative skills that have shown to improve learning and by guiding students’ interaction when meeting impasses, we expect students to engage in a more fruitful collaboration that yields robust learning.

Annotated bibliography

• Diziol, D., Rummel, N., Spada, H., & McLaren, B. M. (submitted). Promoting Learning in Mathematics: Script Support for Collaborative Problem Solving with the Cognitive Tutor Algebra. Submitted to the Conference on Computer Supported Collaborative Learning (CSCL-07). Rutgers University, July 16-21, 2007.
• Walker, E., Rummel, N., McLaren, B. M. & Koedinger, K. R. (submitted). The Student Becomes the Master: Integrating Peer Tutoring with Cognitive Tutoring. Submitted to the Conference on Computer Supported Collaborative Learning (CSCL-07). Rutgers University, July 16-21, 2007.
• Rummel, N., Diziol, D., Spada, H., McLaren, B., Walker, E. & Koedinger, K. (2006) Flexible support for collaborative learning in the context of the Algebra I Cognitive Tutor. Workshop paper presented at the Seventh International Conference of the Learning Sciences (ICLS 2006). Bloomington, IN, USA.
• Walker, E., Koedinger, K. R., McLaren, B. M. and Rummel, N. (2006). Cognitive Tutors as Research Platforms: Extending an Established Tutoring System for Collaborative and Metacognitive Experimentation; In the Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS-2006), Jhongli, Taiwan, June 26-30, 2006.
• Diziol, D. (2006). Development of a collaboration script to improve students` algebra learning when solving problems with the Algebra I, Cognitive Tutor. Diploma Thesis. Albert-Ludwigs-Universität Freiburg, Germany: Institute of Psychology, June 2006.
• McLaren, B., Rummel, N. and others (2005). Improving algebra learning and collaboration through collaborative extensions to the Algebra Cognitive Tutor. Poster presented at the Conference on Computer Supported Collaborative Learning (CSCL-05), May 2005, Taipei, Taiwan.
• Walker, E. (2005). Mutual peer tutoring: A collaborative addition to the Algebra-1 Cognitive Tutor. Paper presented at the 12th International Conference on Artificial Intelligence and Education (AIED-05, Young Researchers Track), July, 2005, Amsterdam, the Netherlands.

References

• Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167-207.
• Berg, K. F. (1993). Structured cooperative learning and achievement in a high school mathematics class. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA.
• Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.
• Kollar, I., Fischer, F., & Hesse, F. W. (in press). Collaboration scripts - a conceptual analysis. Educational Psychology Review.
• Meier, A., Spada, H. & Rummel, N. (in press). A rating scheme for

assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning.

• Rummel, N., & Spada, H. (2005). Learning to Collaborate: An Instructional Approach to Promoting Collaborative Problem Solving in Computer-Mediated Settings. Journal of the Learning Sciences, 14(2), 201-241.
• Teasley, S. D. (1995). The role of talk in children's peer collaborations. Developmental Psychology, 31(2), 207-220.