Walker A Peer Tutoring Addition

Collaborative Extensions to the Cognitive Tutor Algebra: A Peer Tutoring Addition

Erin Walker, Bruce M. McLaren, Nikol Rummel, and Ken Koedinger

Summary Tables

PIs	Bruce M. McLaren, Nikol Rummel, Ken Koedinger
Other Contributers	Graduate Student: Erin Walker Staff: Jonathan Steinhart

Pre Study 1

Study Start Date	05/06
Study End Date	06/06
LearnLab Site	CWCTC
LearnLab Course	Algebra
Number of Students	N = 14
Average # of hours per participant	3 hrs.

Pre Study 2

Study Start Date	11/06
Study End Date	11/06
LearnLab Site	CWCTC
LearnLab Course	Algebra
Number of Students	N = 20
Average # of hours per participant	3 hrs.

Full Study

Study Start Date	04/07
Study End Date	04/07
LearnLab Site	CWCTC
LearnLab Course	Algebra
Number of Students	N = 70
Average # of hours per participant	3 hrs.

Abstract

In this project, the Algebra I Cognitive Tutor is extended to a collaborative learning environment: Instead of the computer tutoring a student, students tutor each other, and the cognitive tutor provides collaborative support. As research has shown, collaborative problem solving and learning has the potential to increase elaboration on the learning content. However, students are not always able to effectively meet the challenges of a collaborative setting. To ensure that students capitalize on collaborative problem solving with the Tutor, a peer tutoring collaboration script was developed that guides their interaction and prompts fruitful collaboration.

In the baseline version of the script (without cognitive tutor support), students in the same class are seated at different computers and take turns tutoring each other. First, they prepare to tutor by solving problems using the regular cognitive tutor. Then, they tutor each other by marking each other right and wrong, monitoring their partner's skills, and give each other hints and feedback. After preparing and tutoring, they reflect on the problems and their interaction, engaging in metacognitive activities. We have expanded the script to include intelligent cognitive support of the peer tutoring, and hope to ultimately expand it to include intelligent collaborative support.

To assess the effectiveness of the script on robust learning, we are conducting a series of in vivo experiments in the Algebra LearnLab. In an initial, small scale study (pre study) that served to establish basic effects and to test the procedure in a classroom setting, we compared peer tutoring with reflection elements to peer tutoring without reflection elements. In a second study that served to explore the effects of cognitive tutoring of peer tutoring, we compared peer tutoring plus cognitive tutoring to peer tutoring. In the full study, we will compare these collaborative conditions to an individual condition to assess the effect of the collaborative Tutor extension to regular Tutor use.

Background and Significance

In our project, we combined two different instructional methods both of which have been shown to improve students’ learning in mathematics: Learning with intelligent tutoring systems (Koedinger, Anderson, Hadley, & Mark, 1997) and collaborative problem solving (Berg, 1993). The Cognitive Tutor Algebra that was used in our study is a tutor for mathematics instruction at the high school level. Students learn with the Tutor during part of the regular classroom sessions. The Tutor's main features are immediate error feedback, the possibility to ask for a hint when encountering impasses, and knowledge tracing, i.e. the Tutor creates and updates a model of the student’s knowledge and selects new problems tailored to the student’s knowledge level. Although several studies have proven its effectiveness, students do not always benefit from learning with the Tutor. First, because the Tutor places emphasis on learning procedural problem solving skills, yet a deep understanding of underlying mathematical concepts is not necessarily achieved (Anderson, Corbett, Koedinger, & Pelletier, 1995). Second, students do not always make good use of the learning opportunities provided by the Cognitive Tutor (e.g. help abuse, see Aleven, McLaren, Roll, & Koedinger, 2004; gaming the system, Baker, Corbett, & Koedinger, 2004). So far, the Cognitive Tutor has been used in an individual learning setting only. However, as research on collaborative learning has shown, collaboration can yield elaboration of learning content (Teasley, 1995), thus this could be a promising approach to reduce the Tutor’s shortcomings. On the other hand, students are not always able to effectively meet the challenges of a collaborative setting (Rummel & Spada, 2005). Collaboration scripts have proven effective in helping people meet the challenges encountered when learning or working collaboratively (Kollar, Fischer, & Hesse, in press).

Combining collaborative activities with intelligent tutoring might combine the benefits of both approaches. Collaborative interaction could augment the effects of an existing ITS by adding deeper interaction, and intelligent tutoring support could improve the quality of student collaboration by providing guidance to students as they attempt to follow a collaboration script. One way to incorporate collaboration into an ITS is to place the intelligent tutoring system in the role of one of the collaborators and have a student interact with this agent. For example, Biswas, Schwartz, Leelawong, Vye, and the TAG-V (2005) have the student collaborate with an agent (“Betty”) in a peer tutoring scenario. The student teaches Betty about ecosystems with the help of a second mentoring agent (“Mr. Davis”), and it has been shown that a student who takes the role of a teacher in this system benefits more than a student who takes the role of a learner in a parallel traditional tutoring system. However, this system does not yet support natural language interaction between the agents involved, and therefore does not take full advantage of important collaborative interaction mechanisms that can increase learning. Another way to combine collaboration and tutoring is to use the ITS to structure and tutor the student collaboration, by connecting collaborative tools with ITS components (see Harrer, McLaren, Walker, Bollen, & Sewall, 2006), and developing a model of good and ineffective collaboration in order to provide on-line hints and feedback (see Soller, 2001). Most research into computer tutoring of the collaboration between two students is still in a preliminary stage.

We integrate collaborative learning with an ITS using a peer tutoring framework. Similar to Biswas et al. (2005), we put the student in the role of the tutor, and intend to have the cognitive elements of the tutoring supported by another agent. However, unlike Biswas et al., (2005), the teachable agent is a real student. Therefore, we have intelligent tutoring components monitoring and providing feedback on student collaboration in addition to student cognition, like Harrer et al. (2006). Our ultimate goal is to allow students to tutor each other through the interface of an ITS, supported by both cognitive and collaborative tutoring.

Glossary

See Peer Tutoring Glossary

Research question

Does collaboration – and in particular scripted collaboration – improve students’ robust learning in the domain of algebra?

Do instructional activities that involve three agents (a student, a peer tutor, and a computer tutor) increase robust learning compared to instructional activities that involve two agents (student and peer tutor, student and computer tutor)?

Independent variables

We vary the agents involved in the interaction using the mode of instruction. There are four conditions:

• Individual condition (Figure 1): Students solved problems with the Algebra I Cognitive Tutor in the regular fashion (i.e. computer program as additional agent).
• Peer tutoring condition (Figure 2): Students tutored each other using the cognitive tutor interface (i.e., another peer as additional agent). They went through two phases:
  • a preparation phase, where students prepared to tutor while solving problems with the Algebra I Cognitive Tutor in the regular fashion
  • a collaboration phase, where students take turns tutoring each other. Peer tutors can mark answers right or wrong, monitor their partner's skills, and give hints and feedback
• Peer tutoring plus reflection: As in the peer tutoring condition, students go through a preparation phase and then tutor each other in the collaboration phase. However, students also go through a series of metacognitive self-reflection activities to encourage them to engage in elaborative interaction
• Peer tutoring plus cognitive tutoring (Figure 3): Students participate in the peer tutoring plus reflection condition, but during the collaboration phase, peer tutors receive cognitive tutor hints and feedback (i.e., a peer and computer program as additional agents). This feedback contains two components:
  • a collaborative prompt that encourages students to interact with each other
  • a cognitive hint targeted at the particular mistake the peer tutee is making

Figure 1. Individual learning in the Cognitive Tutor Algebra.

Figure 2. Peer tutoring in the Cognitive Tutor Algebra. Peer tutor's interface.

Figure 3. Peer tutoring in the Cognitive Tutor Algebra. Adaptive support received by the peer tutor.

Hypothesis

1. We expect the collaborative conditions to outperform the individual conditions in measures of robust learning of the algebra content.

2. We epect the peer tutoring plus cognitive tutoring condition to outperform the peer tutoring alone conditions in measures of robust learning of the algebra content

Dependent variables

• Normal post-test: Students are given a post-test immediately after the study on isomorphic problems
• Far transfer: This paper and pencil test assessed students' understanding of the main mathematical concepts from the learning phase. The transfer items students had to solve tapped the same knowledge components as the problems in instruction, however, the problems where non-isomorphic to those in the instruction, thus demanded students to flexibly apply their knowledge to problems with a new format.
• Accelerated future learning test: Student learning on future equation solving units will be measured.

To compare collaboration skills of students, we will be conducting an analysis of student dialogs during the learning phase.

To assess immediate effects of the instructional variations, we will analyze student progress on training problems as they work through the instruction.

Findings

Pre-study 1:

Our first study compared the peer tutoring to peer tutoring plus reflection conditions. Using pretest and posttest scores, we conducted a two-way (condition x test-time) repeated-measure ANOVA, with test-time as the repeated measure. Posttest scores were significantly higher than pretest scores in both the tutoring and the tutoring+reflection condition (F (1,12) = 15.25, p < .002, η² = 0.56), but there were no significant differences between conditions, and no interaction (see Table 2). To further examine what occurred during the collaboration phase we turned to log data and notes from classroom observation. During peer tutoring, students appeared engaged, and did exhibit many of the positive collaborative behaviors that we were attempting to encourage with our script and that have been shown to correlate with knowledge construction and self-reflection. However, we observed that peer tutors struggled to provide tutees with answers, and did not connect the preparation that they had done with the collaboration phase. For instance, they often did not consult their answer printouts when they did not know the next problem step and thus had to rely on teacher assistance to solve a problem. As a result, tutees skipped problems without completing them correctly. This undesirable behavior differed between the two conditions. Students in the tutoring condition attempted more problems than students in the tutoring+reflection condition, and appeared to complete more problems as well. The average number of problems completed by dyads in the tutoring+reflection condition was low; students in this group took an average of 11 minutes to complete a single problem, compared to a 6 minute average in the tutoring condition. Students in the tutoring condition tended to skip problems they could not solve, completing less than 60% of the problems they attempted. Immediately before skipping a problem, students would generally state their inability to solve it, “I don’t know how to do this one,“ or their lack of motivation, “Just do something and I’ll agree or something.” If students skip problems, they may not learn how to solve difficult problems. However, if they do not complete many problems, they may not be sufficiently exposed to all the skills involved in the unit, and will be given fewer opportunities to master them.

Study 1:

• All students learned as a result of the study. We conducted a two-way (condition x test-time) repeated-measure ANOVA, with test-time (pretest, posttest, or delayed test) as the repeated measure. There was a significant effect for test-time (F(2,72) = 41.303, p < .001), but there were no significant differences between conditions, and no interaction.
• Students in the collaborative conditions solved significantly fewer problems than students in the individual conditions. We conducted a one-way (condition: individual, fixed, adaptive) ANOVA on the number of problems completed per hour in the collaboration phase of the study (see Table 2). For this analysis, we grouped the students in the collaborative conditions by dyad, as the number of problems that one pair member completes (and the time that they take) is dependent on the number of problems the other pair member completes. Condition was indeed significantly related to problems solved (F(2,34) = 8.764, p = .001).
• There were some interesting lacks of difference in tutee and tutor behaviors within problems. For example, across all conditions, tutees tended to make the same number of mistakes per problem, request the same amount of help per problem, and receive the same amount of help from their tutors per problem
• There was an apparent relationship between tutee impasses and tutor learning. For example, the collaborative conditions differed on how easy it was for students to move to the next problem. In the adaptive condition, students could not continue unless they had successfully completed the problem, making it possible for students to get “stuck”, where they repeatedly tried incorrectly to move to the next problem. The number of these incorrect done tries was negatively correlated with tutee gain scores on the delayed test (r = -.591, p = .056), but positively correlated with tutor gain scores on the delayed test (r = .463, p = .115). In the fixed condition, students were not notified when their attempts to continue were incorrect, and thus could “skip” to the next problem even if the previous problem was not done. Problems skipped were negatively correlated with tutee learning (r = -.614, p = .059) and tutor learning (r = -.369, p = .329). If problems were skipped tutors did not benefit from tutee impasses.

Explanation

This study is part of the Interactive Communication cluster, and its hypothesis is a specialization of the IC cluster’s central hypothesis. According to the IC cluster’s hypothesis, instruction that yields robust learning should be designed to have the right target paths. Furthermore, it should improve the students’ path choice in the learning event space, i.e., increasing the probability that students take correct paths and decreasing the probability that students take alternative paths.

By enhancing the Cognitive Tutor with collaboration, the study tried to reach these two goals in two steps:

• Individual condition: The existing Cognitive Tutor already offers correct learning paths. However, individual learning on the Cognitive Tutor has several shortcomings. For example, students don’t have the opportunity to reflect on the underlying mathematical concepts in natural language interaction in order to gain a deeper understanding. In addition, students do not always use the offered learning paths (as the opportunity to reflect on on-demand hints and error feedback) effectively, thus often do not engage in sense making processes
• Peer tutoring condition (1st step): By enhancing the Algebra I Cognitive Tutor to be a collaborative learning environment, we added a learning resource – the learning partner. This adds further correct learning paths to the learning event space, for instance, learning by giving explanations, the possibility to request help, and learning by knowledge co-construction. However, similarly to the learning paths in an individual setting, students do not always capitalize on these learning opportunities. Further, if students lack the necessary expertise to tutor, they will not master the desired knowledge components, even if they take a correct learning path.
• Peer plus cognitive tutoring condition (2nd step): To increase the probability that students are mastering the correct knowledge components, the cognitive tutor helps peer tutors interact with their partners. Students use both the peer tutor and the cognitive tutor as learning resources.

Annotated bibliography

• Walker, E., Rummel, N., and Koedinger, K. R. To Tutor the Tutor: Adaptive Domain Support for Peer Tutoring. To appear at the 9th International Conference on Intelligent Tutoring Systems. 2008.
• Walker, E., McLaren, B. M., Rummel, N., and Koedinger, K. R. Who Says Three's a Crowd? Using a Cognitive Tutor to Support Peer Tutoring. To appear in the Proceedings of the 13th International Conference on Artificial Intelligence and Education. 2007.
• Walker, E., Rummel, N., McLaren, B. M. & Koedinger, K. R. The Student Becomes the Master: Integrating Peer Tutoring with Cognitive Tutoring. Short paper at the Conference on Computer Supported Collaborative Learning (CSCL-07). Rutgers University, July 16-21, 2007.
• Walker, E., Koedinger, K., McLaren, B. M., & Rummel, N. (2006). Cognitive tutors as research platforms: Extending an established tutoring system for collaborative and metacognitive experimentation. Lecture Notes in Computer Science, Volume 4053/2006. Proceedings of the 8th International Conference on Intelligent Tutoring Systems (pp. 207-216). Berlin: Springer
• Walker, E. (2005). Mutual peer tutoring: A collaborative addition to the Algebra-1 Cognitive Tutor. Paper presented at the 12th International Conference on Artificial Intelligence and Education (AIED-05, Young Researchers Track), July, 2005, Amsterdam, the Netherlands.

References

• Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modelling to meta-cognitive skills. In J. C. Lester, R. M. Vicari & F. Paraguaçu (Eds.), Proceedings of Seventh International Conference on Intelligent Tutoring Systems, ITS 2004 (pp. 227-239). Berlin: Springer.
• Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167-207.
• Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004). Detecting student misuse of intelligent tutoring systems. Paper presented at the Proceedings of the 7th International Conference on Intelligent Tutoring Systems.
• Berg, K. F. (1993). Structured cooperative learning and achievement in a high school mathematics class. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA.
• Biswas, G., Schwartz, D. L., Leelawong, K., Vye, N., & TAG-V. (2005). Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence, 19, 363–392.
• Carmien, S., Kollar, I., Fischer, G. & Fischer, F. (2006). The interplay of internal and external scripts. In F. Fischer, I. Kollar, H. Mandl &, J. Haake, Scripting computer-supported communication of knowledge. Cognitive, computational, and educational perspectives (pp. 289-311). New York: Springer.
• Fantuzzo, J. W., Riggio, R. E., Connelly, S., & Dimeff, L. A. (1992). Effects of reciprocal peer tutoring on mathematics and school adjustment: A componential analysis. Journal of Educational Psychology, 84(3), 331-339.
• Fuchs, L.S., Fuchs, D., Prentice, K., Burch, M., Hamlett, C.L., Owen, R., & Schroeter, K. (2003). Enhancing third-grade students' mathematical problem solving with self-regulated learning strategies. Journal of Educational Psychology, 95(2), 306-315.
• Harrer, A., McLaren, B. M., Walker, E., Bollen, L., and Sewall, J. (2006). Creating Cognitive Tutors for Collaborative Learning: Steps Toward Realization. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI) (2006) 16: 175-209.
• Johnson, D. W. and Johnson, R. T. (1990). Cooperative learning and achievement. In S. Sharan (Ed.), Cooperative learning: Theory and research (pp. 23-37). New York: Praeger.
• King, A., Staffieri, A., & Adelgais, A. (1998). Mutual peer tutoring: Effects of structuring tutorial interaction to scaffold peer learning. Journal of Educational Psychology, 90, 134-152.
• Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.
• Kollar, I., Fischer, F., & Hesse, F. W. (in press). Collaboration scripts - a conceptual analysis. Educational Psychology Review.
• Ploetzner, R., Dillenbourg, P., Preier, M.,&Traum, D. (1999). Learning by explaining to oneself and to others. In P. Dillenbourg (Ed.), Collaborative learning. Cognitive and computational approaches (pp. 103–121). Amsterdam: Pergamon.
• Ritter, S., Blessing, S. B., & Hadley, W. S. (2002). SBIR Phase I Final Report 2002. Department of Education. Department of Education RFP ED: 84-305S.
• Rummel, N., & Spada, H. (2005). Learning to Collaborate: An Instructional Approach to Promoting Collaborative Problem Solving in Computer-Mediated Settings. Journal of the Learning Sciences, 14(2), 201-241.
• Soller, A.L. (2001). Supporting Social Interaction in an Intelligent Collaborative Learning System. International Journal of Artificial Intelligence in Education, 12, 40-62.
• Teasley, S. D. (1995). The role of talk in children's peer collaborations. Developmental Psychology, 31(2), 207-220.
• Webb, N.M., Troper, J.D., & Fall, R. (1995). Constructive activity and learning in collaborative small groups. Journal of Educational Psychology, 87, 406-423.

Connections

Like this study, Rummel Scripted Collaborative Problem Solving adds scripted collaborative problem solving to the Cognitive Tutor Algebra. The studies differ in the way collaboration is integrated in the Tutor. First, in the Rummel et al. study, both students first prepare one subtasks of a problem to mutually solve the complex story problem later on. Thus, although the students are experts for different parts of the problem, they have a comparable knowledge level during collaboration. In contrast, in this study, one student prepares to teach his partner. Then, they change roles. Thus, during collaboration, their knowledge level differs. Second, in the Rummel et al. study, collaboration was face to face, whereas this study used a chat tool for interaction.

Similar to the adaptive script component of the Collaborative Problem-Solving Script, the Help Tutor project aims at improving students' help-seeking behavior and at reducing students' tendency to game the system.
Furthermore, both studies contain instructions to teach metacognition. The metacognitive component in our study instructs students to monitor their interaction in order to improve it in subsequent collaborations; the Help Tutor project asks students to evaluate their need for help in order to improve their help-seeking behavior when learning on the Tutor.

Both in this study and in the Reflective Dialogue study from Katz, students are asked to engage in reflection following each problem-solving. In this study, the reflection concentrates on the collaborative skills, while in Katz' study, the reflection concentrates on students' domain knowledge of the main principles applied in the problem.

Furthermore, both our study and the Help Lite (Aleven, Roll) aim at improving conceptual knowledge.

Future plans

Our future plans for June 2007 - December 2007:

• Run full study
• Analyze process data
• Conference contributions at CSCL and AIED
• Paper publication