Rummel Scripted Collaborative Problem Solving

Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving

Nikol Rummel, Dejana Diziol, Bruce M. McLaren, and Hans Spada

Summary Tables

PIs	Robert G.M. Hausmann & Kurt VanLehn
Other Contributers	Brett van de Sande & Robert Shelby

Pilot Study (Lab)

Study Start Date	02/11/06
Study End Date	02/16/06
LearnLab Site	None (lab study)
LearnLab Course	Algebra
Number of Students	N = 5
Average # of hours per participant	3 hrs.

Pre Study

Study Start Date	03/13/06
Study End Date	03/21/06
LearnLab Site	CWCTC
LearnLab Course	Algebra
Number of Students	N = 30
Average # of hours per participant	3 hrs.

Full Study

Study Start Date	04/03/06
Study End Date	04/07/06
LearnLab Site	CWCTC
LearnLab Course	Algebra
Number of Students	N = 139
Average # of hours per participant	3 hrs.

Abstract

In this project, the Algebra I Cognitive Tutor is extended to a collaborative learning environment: students learn to solve system of equations problems while working in dyads. As research has shown, collaborative problem solving and learning has the potential to increase elaboration on the learning content. However, students are not always able to effectively meet the challenges of a collaborative setting. To ensure that students capitalize on collaborative problem solving with the Tutor, a collaboration script was developed that guides their interaction and prompts fruitful collaboration. During the scripted problem solving, students alternate between individual and collaborative phases. Furthermore, they reflect on the quality of their interaction during a recapitulation phase that follows each problem they solve. During this phase, students self evaluate their interaction and plan how to improve it, thus they engage in metacognitive activities.

To assess the effectiveness of the script, we conducted two classroom studies. In an initial, small scale study (pre study) that served to establish basic effects and to test the procedure in a classroom setting, we compared scripted collaboration with an unscripted collaboration condition in which students collaborated without support. In the full study, we furthermore compared these collaborative conditions to an individual condition to assess the effect of the collaborative Tutor extension to regular Tutor use. The experimental learning phase took place on two days of instruction. On the third day, during the test phase, students completed several post tests that assessed robust learning. Analyses of the collected measures are in progress. Results from one test of the pre study revealed significant differences between conditions. This test assessed students' transfer of main mathematical concepts of the learning content to tasks of a different format. These differences where particularly clear for the open format questions that asked students to explain their problem-solving. Further results will be presented soon.

Background and Significance

In our project, we combined two different instructional methods both of which have been shown to improve students’ learning in mathematics: Learning with intelligent tutoring systems (Koedinger, Anderson, Hadley, & Mark, 1997) and collaborative problem solving (Berg, 1993). The Cognitive Tutor Algebra that was used in our study is a tutor for mathematics instruction at the high school level. Students learn with the Tutor during part of the regular classroom sessions. The Tutor's main features are immediate error feedback, the possibility to ask for a hint when encountering impasses, and knowledge tracing, i.e. the Tutor creates and updates a model of the student’s knowledge and selects new problems tailored to the student’s knowledge level. Although several studies have proven its effectiveness, students do not always benefit from learning with the Tutor. First, because the Tutor places emphasis on learning procedural problem solving skills, yet a deep understanding of underlying mathematical concepts is not necessarily achieved (Anderson, Corbett, Koedinger, & Pelletier, 1995). Second, students do not always make good use of the learning opportunities provided by the Cognitive Tutor (e.g. help abuse, see Aleven, McLaren, Roll, & Koedinger, 2004; gaming the system, Baker, Corbett, & Koedinger, 2004). So far, the Cognitive Tutor has been used in an individual learning setting only. However, as research on collaborative learning has shown, collaboration can yield elaboration of learning content (Teasley, 1995), thus this could be a promising approach to reduce the Tutor’s shortcomings. On the other hand, students are not always able to effectively meet the challenges of a collaborative setting (Rummel & Spada, 2005). Collaboration scripts have proven effective in helping people meet the challenges encountered when learning or working collaboratively (Kollar, Fischer, & Hesse, in press). To ensure that students would profit from a collaboratively enhanced Tutor environment, we thus developed a collaboration script that prompts fruitful interaction on the Tutor while students learned to solve “systems of equations”, a content novel to the participating students. The script served two goals: First, to improve students’ robust learning of knowledge components relevant for the systems of equation units (i.e. both problem solving skills and understanding of the concepts). Second, students should increase their explicit knowledge of what constitutes a fruitful collaboration and improve their collaboration skills in subsequent interactions without further script support. This is particularly important due to the risk of overscripting collaboration that has been discussed in conjunction with scripting for longer periods of time (Rummel & Spada, 2005).

Glossary

See Scripted Collaborative Problem Solving Glossary

Research question

Does collaboration – and in particular scripted collaboration – improve students’ robust learning in the domain of algebra?

Does the script approach improve students’ collaboration, and does this result in more robust learning of the algebra content?

Independent variables

The independent variable was the mode of instruction. Three conditions were implemented in the learning phase that took place on two days of instruction:

• Individual condition: Students solved problems with the Algebra I Cognitive Tutor in the regular fashion (i.e. computer program as additional agent).
• Unscripted collaborative condition: Students solved problems on the Cognitive Tutor while working in dyads; thus, a second learning resource was added to the regular Tutor environment (i.e., both a peer and a computer program as additional agents).
• Scripted collaboration condition: As in the unscripted collaborative condition, students solved problems on the Tutor while working in dyads; however, their dyadic problem solving was guided by the Collaborative Problem-Solving Script (CPS) (i.e., both a peer and a computer program as additional agents). Three script components supported their interaction in order to reach a more fruitful collaboration and improved learning:
  • A fixed script component structured students’ collaboration in an individual and a collaborative phase. During the individual phase, each student solved a single equation problem; during the collaborative phase, these problems were combined to a more complex system of equations problem. This division of expertise aimed at increasing the individual's responsibility for the common problem-solving process and thus at yielding a more fruitful interaction. In addition, the fixed script component structured the collaborative phase in several sub steps and prompted students to engage in elaborative interaction.
  • An adaptive script component provided prompts to improve the collaborative learning process when the dyad encountered difficulties in their problem solving. E.g., to prevent help abuse in the tutoring system and to stop students from proceeding in the hint sequence until reaching the bottom out hint, the dyad is prompted to elaborate on the Tutor hints when reaching the penultimate hint of the hint sequence. This aims at enabling students to find the answer by themselves on basis of the help received and at students gaining knowledge to solve similar steps later on their own.
  • A metacognitive script component prompted students to engage in reflection of the group process to improve their collaboration over subsequent interactions. Following each collaborative phase, students rated their interaction on 8 collaboration skills and set goals for the following collaboration. This phase was referred to as the "recapitulation phase"

In all conditions, the Cognitive Tutor supported students’ problem solving by flagging their errors and by providing on-demand hints.

Communication between the collaborative partners in the collaborative conditions was face to face: they joined together at a single computer to solve the problems. In the scripted collaboration condition, the script structured the allocation of subtasks between the two students.

In the pre study that aimed at establishing basic effects and at testing the procedure in a classroom setting, we only compared unscripted and scripted collaborative problem solving.

Hypothesis

The hypothesis can be stated in two parts: knowledge gains in the domain of algebra and improvement of the dyads’ collaboration skills:

1. We expect the collaborative conditions – in particular the scripted collaboration condition – to outperform the individual conditions in measures of robust learning of the algebra content.
2. We expect that problem-solving with the Collaborative Problem-Solving Script will lead to a more effective interaction than unscripted collaboration in the learning phase. In addition, we expect that the newly gained collaborative skills will sustain during subsequent collaborations in the test phase when script support is no longer available.

Dependent variables

Students took several post tests during the test phase that took place two days after instruction:

• Near transfer, immediate: Student progress on training problems during the learning phase was analyzed. Their improvement in solving system of equations, i.e. their ability to transfer from one problem to an isomorphic problem with different content, can be assessed by looking at the learning curves.
• Near transfer, retention: Two post-tests measured near transfer. As during the learning phase, students solved system of equations problems on the Cognitive Tutor. The problems were “isomorphic” to those in the instruction, but had a different content. One near transfer test was solved collaboratively to measure both students’ algebra skills and their collaboration skills, the other near transfer test was solved individually to assess if the students were able to transfer the gained knowledge to isomorphic problems when working on their own.
• Far transfer: This paper and pencil test assessed students' understanding of the main mathematical concepts from the learning phase: slope, y-intercept and intersection point. The conceptual tasks students had to solve tapped the same knowledge components as the problems in instruction; however, the problems where non-isomorphic to those in the instruction, thus demanded students to flexibly apply their knowledge to problems with a new format. The test consisted of two problem sets with two different question formats: problem set 1 asked for the basic concepts y-intercept and slope, problem set 2 asked for the novel system’s concept of intersection point. Both sets were composed of questions with a distinct answer format and open format questions that asked students to explain their problem-solving.
• Accelerated future learning: To measure accelerated future learning, students learned how to solve problems in a new area of algebra – inequality problems. This test was administered on the Cognitive Tutor, thus students could ask the Tutor for hints and received immediate error feedback. According to their condition during the experimental learning phase, they either worked individually or collaboratively on this post test. However, in contrast to the learning phase, they did not receive additional instructions to support their collaboration.

Several variables from these post tests are compared to test the two hypotheses.

1. For the post tests conducted on the Cognitive Tutor (retention and accelerated future learning), variables such as number of errors per problem, time per problem, decrease of error rates over the course of several problems etc are extracted from the log files to compare conditions.
   For the paper and pencil post test (far transfer), we developed a scoring systems. For distinct format questions, students received 0 points for incorrect/missing answers and 1 point for correct answers; for open format questions , students received 0 points for incorrect, irrelevant or missing answers, 1 point for partly elaborated answers, and 2 points for fully elaborated answers. We summed points separately for the two question formats and the two problem sets of this post-test, resulting in four variables.
2. To compare collaboration skills of students, we recorded students’ dialogues both during learning phase and during test phase. Based on a rating scheme from one of our former projects (Meier, Spada, & Rummel, 2007), we develop a rating scheme to assess students’ interaction on skill dimensions such as “giving elaborated explanations to the partner”.
   Furthermore, the paper and pencil post test entailed an open format question asking students to explain what is important for a fruitful collaboration. The answers to this question are also coded and compared between conditions

Method

The study procedure took place during 3 classroom periods over the course of a week. During day 1 and day 2 (the learning phase), the experimental conditions (i.e. the independent variables) were implemented. On day 3, students had to solve several post-tests that assessed different aspects of robust learning.

Two in vivo classroom studies were conducted: a pre study with 3 classrooms, and a full study with 8 classrooms. The pre study aimed at establishing basic effects of the script and testing the procedure in a classroom setting, thus we only compared scripted to unscripted collaboration. In the full study, we compared the scripted collaboration condition to both an individual condition and an unscripted collaboration to evaluate the merits of both enhancing the Tutor with collaboration in general and scripted collaboration in particular. Due to the disruptiveness of students in the same class using different interventions, we used a between-class design.

Participants pre study:

The unscripted condition consisted of two classes (12 and 4 students), and the scripted condition consisted of one class (13 students). All classes were taught by the same teacher.

Participants full study:

• Individual condition: 1 class (21 students)
• Unscripted condition: 3 classes (18 + 16 + 16 dyads = 25 dyads)
• Scripted condition: 4 classes (16 + 16 + 18 + 18 dyads = 34 dyads)

Findings

Findings pre study:

First results of the paper and pencil test (far transfer) revealed substantial differences between conditions. The analysis was restricted to students who always worked collaboratively when present, as we are interested in the script’s effect on collaborative learning in particular. Due to student absenteeism, only 9 students in the unscripted and 10 students in the scripted condition were included in our analysis.

Significant differences were found both in the MANOVA (Pillai-Spur, F(4, 14) = 7.35, p < .05) and in subsequent ANOVAs. Multiple choice answers on basic concepts did not show a significant difference, F(1,17) = 2.26, ns. However, the scripted condition outperformed the unscripted condition on the discrete answer questions about the system’s concept, F(1,17) = 22.16, p < .01. A significant difference between conditions was also found for the open format questions of both problem sets with F(1,17) = 5.85, p < .05 for the basic concepts and F(1,17) = 17.01, p < .01 for the system’s concept. Particularly the results of the open format questions demonstrate the script’s effect on conceptual knowledge acquisition: after scripted interaction during the learning phase, students were better at articulating their mathematical thinking compared to their unscripted counterparts. However, it should be noted that in general students in both conditions had difficulties providing explanations, thus only reached low scores in the open format questions. The amount of wrong explanations and the number of students who did not even try to articulate their thinking was very high.

Results on tests that were administered on the Cognitive Tutor (hypothesis 1: near transfer, immediate; near transfer, retention; acceleration of future learning) as well as results of the analysis of the collaboration process (hypothesis 2) are coming soon.

Findings full study:

Coming soon.

Explanation

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. According to the IC cluster’s hypothesis, instruction that yields robust learning should be designed to have the right target paths. Furthermore, it should improve the students’ path choice in the learning event space, i.e., increasing the probability that students take correct paths and decreasing the probability that students take alternative paths.

By enhancing the Cognitive Tutor with collaboration, the study tried to reach these two goals in two steps:

• Individual condition: The existing Cognitive Tutor already offers correct learning paths. However, individual learning on the Cognitive Tutor has several shortcomings. For example, students don’t have the opportunity to reflect on the underlying mathematical concepts in natural language interaction in order to gain a deeper understanding. In addition, students do not always use the offered learning paths (as the opportunity to reflect on on-demand hints and error feedback) effectively, thus often do not engage in sense making processes
• Unscripted collaboration condition (1st step): By enhancing the Algebra I Cognitive Tutor to be a collaborative learning environment, we added a second learning resource – the learning partner. This adds further correct learning paths to the learning event space, for instance, learning by giving explanations, the possibility to request help, and learning by knowledge co-construction. However, similarly to the learning paths in an individual setting, students do not always capitalize on these learning opportunities.
• Scripted collaboration condition (2nd step): To increase the probability that students take correct learning paths, students’ interaction in the scripted collaboration condition is guided by the Collaborative Problem Solving Script. By prompting collaboration skills that have shown to improve learning and by guiding students’ interaction when meeting impasses, we expect students to engage in a more fruitful collaboration that yields robust learning.

Annotated bibliography

• Rummel, N., Diziol, D., Spada, H., & McLaren, B. M. (submitted). Scripting Collaborative Problem Solving with the Cognitive Tutor Algebra: A Way to Promote Learning in Mathematics. Submitted to the European Association for Research on Learning and Instrution (EARLI-07). Budapest, August 28 - September 1, 2007.
• Rummel, N., Spada, H., Diziol, D. (submitted). Evaluating Collavborative Extensions to the Cognitive Tutor Algebra in an In Vivo Experiment. Lessons Learned. Submitted to the European Association for Research on Learning and Instrution (EARLI-07). Budapest, August 28 - September 1, 2007.
• Diziol, D., Rummel, N., Spada, H., & McLaren, B. M. (submitted). Promoting Learning in Mathematics: Script Support for Collaborative Problem Solving with the Cognitive Tutor Algebra. Submitted to the Conference on Computer Supported Collaborative Learning (CSCL-07). Rutgers University, July 16-21, 2007.
• Rummel, N., Diziol, D., Spada, H., McLaren, B., Walker, E. & Koedinger, K. (2006) Flexible support for collaborative learning in the context of the Algebra I Cognitive Tutor. Workshop paper presented at the Seventh International Conference of the Learning Sciences (ICLS 2006). Bloomington, IN, USA.
• Diziol, D. (2006). Development of a collaboration script to improve students` algebra learning when solving problems with the Algebra I, Cognitive Tutor. Diploma Thesis. Albert-Ludwigs-Universität Freiburg, Germany: Institute of Psychology, June 2006.
• McLaren, B., Rummel, N. and others (2005). Improving algebra learning and collaboration through collaborative extensions to the Algebra Cognitive Tutor. Poster presented at the Conference on Computer Supported Collaborative Learning (CSCL-05), May 2005, Taipei, Taiwan.

References

• Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modelling to meta-cognitive skills. In J. C. Lester, R. M. Vicari & F. Paraguaçu (Eds.), Proceedings of Seventh International Conference on Intelligent Tutoring Systems, ITS 2004 (pp. 227-239). Berlin: Springer.
• Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167-207.
• Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004). Detecting student misuse of intelligent tutoring systems. Paper presented at the Proceedings of the 7th International Conference on Intelligent Tutoring Systems.
• Berg, K. F. (1993). Structured cooperative learning and achievement in a high school mathematics class. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA.
• Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.
• Kollar, I., Fischer, F., & Hesse, F. W. (in press). Collaboration scripts - a conceptual analysis. Educational Psychology Review.
• Meier, A., Spada, H. & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning. published online first at DOI http://dx.doi.org/10.1007/s11412-006-9005-x, February 2007
• Rummel, N., & Spada, H. (2005). Learning to Collaborate: An Instructional Approach to Promoting Collaborative Problem Solving in Computer-Mediated Settings. Journal of the Learning Sciences, 14(2), 201-241.
• Teasley, S. D. (1995). The role of talk in children's peer collaborations. Developmental Psychology, 31(2), 207-220.

Connections

As this study, Walker A Peer Tutoring Addition adds scripted collaborative problem solving to the Cognitive Tutor Algebra. The studies differ in the way collaboration is integrated in the Tutor. First, in this study, both students first prepare one subtasks of a problem to mutually solve the complex story problem later on. Thus, although the students are experts for different parts of the problem, they have a comparable knowledge level during collaboration. In contrast, in Walker et al.'s study, one student prepares to teach his partner. Then, they change roles. Thus, during collaboration, their knowledge level differs. Second, in this study, collaboration was face to face, whereas the Walker et al. study used a chat tool for interaction.

Similar to the adaptive script component of the Collaborative Problem-Solving Script, the Help Tutor project aims at improving students' help-seeking behavior and at reducing students' tendency to game the system.
Furthermore, both studies contain instructions to teach metacognition. The metacognitive component in our study instructs students to monitor their interaction in order to improve it in subsequent collaborations; the Help Tutor project asks students to evaluate their need for help in order to improve their help-seeking behavior when learning on the Tutor.

Both in this study and in the Reflective Dialogue study from Katz, students are asked to engage in reflection following each problem-solving. In this study, the reflection concentrates on the collaborative skills, while in Katz' study, the reflection concentrates on students' domain knowledge of the main principles applied in the problem.

Furthermore, both our study and the Help Lite (Aleven, Roll) aim at improving conceptual knowledge.

Study