Difference between revisions of "Composition Effect Kao Roll"

The Composition Effect - What is the Source of Difficulty in Problems which Require Application of Several Skills?

Ido Roll, Yvonne Kao, Kenneth E. Koedinger

Meta-data

PI's: Ido Roll, Yvonne Kao, Kenneth E. Koedinger

Other Contributers:

Study # Start Date End Date LearnLab Site # of Students Total Participant Hours DataShop?
1 9/2006 10/2006 CWCTC (Algebra) 98 98 yes

Abstract

This study found that the presence of distracters creates significant difficulty for students solving geometry area problems, but that practice on composite area problems improves students’ ability to ignore distracters. In addition, this study found some support for the hypothesis that the increased spatial processing demands of a complex diagram can negatively impact performance and could be a source of a composition effect in geometry.

Glossary

- Composite problems: Problems which require the application of several skills, such as solving 3x+6=0 for x.

- Single-step problems: Problems which require the application of a single skill, such as y+6=0 or 3x=-6

- DFA (Difficulty Factor Analysis): A test that includes pairs of items varying along one dimension only. It allows to evaluate the difficulty level of the single dimensions along which the problems differ.

- The Composition Effect: The effect according to which composite problems are harder than a set of single-step problems using the same skills.

Research question

What is the main source of difficulty in composite problems?

Independent Variables

An instruction in the form of solved-example, targeting a common misconception - identifying base and hight in a cluttered environment.

Dependent variables

Three tests are used in the study: - Pre-test: given before all instruciton - Mid-test: given after students learned about single-step problems and before composite problems - Post-test: after students have learned and practice all material.

The tests include the following items. Some of which are transfer items, evaluating robust learning, since they require and adaptive application of the knowledge learned and practiced in class.

• Simple diagram:
1. no distractors, canonical orientation
2. distractors, canonical orientation
3. no distractors, tilted orientation
4. distractors, tilted orientation
• Complex diagram:
1. Given complex diagram, ask for skill A
2. Given complex diagram, ask for skill B
3. Given steps A and B, ask for skills C (which requires A and B)
4. Given complex diagram, ask for C (which requires A and B)

Hypothesis

1. Adding distracters to a basic area problem and rotating the figure from its standard orientation will make the problem more difficult.
2. We will find a composition effect in area problems, in that the probability of correctly completing a composite problem is not equal to the product of the probabilities of correctly completing the three subskills: P(Composite) ≠ P(Outer) × P(Inner) × P(Subtract).
3. P(Composite) > P(Outer) × P(Inner) × P(Subtract), P(Outer) < P(Simple Outer Eq.), and P(Composite) = P(Simple Outer Eq.) × P(Inner) × P(Subtract) due to the demands of spatially parsing the diagram.

Findings

An alpha value of .05 was used for all statistical tests.

Comparison of Mid-test and Post-test Performance

Scores on the pretest were at floor, ranging from 0 to 50% correct (M = 14.94%, SD = 13.61%). Pre-test scores did not correlate significantly with either mid-test scores or post- test scores. Thus we did not analyze the pretest further. Performance on the mid-test and the post-test were significantly correlated (r2 = 0.239, p < .001). A paired t- test found significant gains in overall performance from mid-test to post-test, t(65) = 3.115, p = 0.003, 2-tailed, with highly significant gains on Simple problems, t(65) = 3.104, p = 0.003, 2-tailed, and significant gains on Complex problems, t(65) = 2.308, p = 0.024. Participants performed better on the Simple problems than on the Complex problems. This difference was significant at mid-test, t(65) = 2.214, p = .030, 2-tailed, and at post-test, t(65) = 2.355, p = .022, two-tailed. These results are presented in Table 1.

Effects of Distracters, Orientation, and Shape on Simple Problem Performance

We used a binary logistic regression analysis to predict the probability that students would answer a Simple problem correctly on the mid-test and the post-test. Distracters, Rotation, and three dummy variables coding diagram shape—parallelogram, pentagon, trapezoid, or triangle - were entered into the equation as predictors.

Table 1: Mean performance on mid-test and post-test by diagram type1

Mid-test (%) Post-test (%) Gain (%)
Type M SD M SD M SD
Overall 65.34 27.79 75.41 23.66 10.07** 26.26
Simple 68.56 30.79 79.17 23.03 10.61** 27.76
Complex 61.74 30.13 71.21 31.08 9.47* 33.33
Simple-Complex 6.82* 25.03 7.96* 27.44

At mid-test, the full model differed significantly from a model with intercept only, χ2 (5, N = 264) = 17.884, Nagelkerke R2 = .092, p = .003. Overall, this model had a success rate of 69.3%, correctly classifying 90.1% of correct responses and 24.1% of incorrect responses. The presence of distracters was a significant predictor of problem accuracy at mid-test. Students were only .541 times as likely to respond correctly when the problem contained distracters. Shape was also a significant predictor. Students were only .484 times as likely to respond correctly when the problem involved a pentagon over a triangle. Rotation was not a significant predictor of problem accuracy. At post-test, the full model differed significantly from a model with intercept only, χ2 (5, N = 264) = 15.533, Nagelkerke R2 = .089, p = .008. Overall, this model had a success rate of 79.2%, correctly classifying 100% of the correct responses, but 0% of incorrect responses. Distracters were no longer a significant predictor at post- test, and Rotation remained a non-significant predictor. Shape remained a significant predictor of problem accuracy. Students were only .335 times as likely to respond correctly when the problem involved a pentagon over a triangle.

Effects of Skill Composition and Shape on Complex Problem Performance

We used a binary logistic regression analysis to predict the probability that students would answer a Complex problem correctly on the mid-test and the post-test. Whether the problem required an Outer calculation, an Inner calculation, or Subtraction were entered into the equation as predictors. As before, three dummy variables coding diagram shape were entered as well. At mid-test, the full model differed significantly from a model with intercept only, χ2 (6, N = 264) = 12.862, Nagelkerke R2 = .065, p = .045. Overall, this model had a success rate of 64.0%, correctly classifying 85.9% of correct responses and 28.7% of incorrect responses. At post-test, the full model also differed significantly from a model with intercept only, χ2 (6, N = 264) = 25.019, Nagelkerke R2 = .129, p < .001. Overall, this model had a success rate of 70.5%, correctly classifying 95.7% of correct responses and 7.9% of incorrect responses. In both models, the Outer calculation was the only significant predictor of a correct response. When the problem required an Outer calculation—in the Outer and Composite conditions—students were only .439 and .258 times as likely to respond correctly at mid-test and post-test, respectively.

Predictors of Performance on Composite Problems.

We used a binary logistic regression to predict the probability that a student would answer a Composite Complex problem correctly on the mid-test and the post-test, given his/her success on: Inner, Outer, and Subtract, the Distracters+Rotation (DR) Simple problem that is mathematically-equivalent to Outer; and the shape of the diagram—parallelogram, pentagon, trapezoid, or triangle— coded using three dummy variables. This model differed significantly from a model with intercept only for both mid- test, χ2 (7, N = 66) = 26.567, Nagelkerke R2 = .442, p < .001, and post-test, χ2 (7, N = 66) = 30.466, Nagelkerke R2 = .495, p < .001. The mid-test model had overall success rate of 75.8%, correctly classifying 75.0% of correct responses and 76.5% of incorrect responses. The post-test model had an overall success rate of 77.3%, correctly classifying 83.8% of correct responses and 69.0% of incorrect responses. The significant predictor variables in the model changed from mid-test to post-test. At mid-test, Subtract and DR success were significant predictors of a correct response on Composite problems. Students who answered Subtract correctly were 9.168 times more likely to answer Composite correctly. Students who answered DR correctly were 5.891 times more likely to answer Composite correctly. At post- test, Subtract and DR success remained significant predictors with odds ratios of 20.532 and 9.277, respectively. In addition, Outer success became a significant predictor. Students who answered Outer correctly were 6.366 times more likely to answer Composite correctly. Shape was not a significant predictor at either mid-test or post-test. These results are presented in Table 1.

Assessing the Difficulty of Spatial Parsing

We took Accuracy(Outer) × Accuracy(Inner) × Accuracy(Subtract) and compared this to Accuracy(Composite) for each student using a paired, 2-tailed t-test. This difference was significant at mid-test, t(65) = 2.193, p = .032. Students performed better on Composite (M = 48.48%, SD = 50.36%) than was predicted by the product (M = 33.33%, SD = 47.52%). This difference was no longer significant at post- test. In contrast, Accuracy(DR) × Accuracy(Inner) × Accuracy(Subtract) did not differ significantly from Accuracy(Composite) at either mid-test or post-test. Paired, 2-tailed t-tests found that performance did not differ significantly between the Outer and DR at either mid-test, t(65) = -.851, p = .398, or post-test, t(65) = -1.356, p = .180.

Explanation

We will return to our original hypotheses to begin the discussion. It was clear that distracters had a negative impact on Simple performance at mid-test, although this effect had largely disappeared by post-test. Although we did not find significant effects of Rotation on Simple performance, we did find evidence that many students simply rotated the paper until they were viewing the figure in a standard orientation, effectively negating our Rotation manipulation. Thus, our first hypothesis is partially supported. We did find a composition effect in area problems, and the probability of success on Composite problems could not be predicted by simply multiplying the probabilities of success for the three subproblems. Thus, our second hypothesis is supported. We only found partial support for our hypothesis that the source of the composition effect is due to the diagram parsing load. We did find that the probability of success on Composite problems was greater than the product of probabilities for the three subproblems, but only at mid-test. In addition, there were no significant differences between Outer and DR at either mid-test or post-test, although we feel it is worth noting that the data are trending in the predicted direction, with Outer being more difficult than DR on both mid-test and post-test. Finally, our P(Composite) = P(Simple Outer Eq.) × P(Inner) × P(Subtract) model did a good job of predicting actual performance on Composite problems. To conclude our discussion, we would like to address the differences between performance at mid-test and performance at post-test and the implications for designing instructional interventions. First, we would like to note that instruction in the Area Composition unit of the Geometry Cognitive Tutor was able to improve performance on all skills, not just skills new to composite problems. This suggests that students may not have fully mastered the single-step area skills prior to beginning Area Composition, but that Area Composition continues to provide practice on these skills. Furthermore, the single-step skill practice in the Area Composition unit seems particularly effective at removing the effects of distracters on performance. This makes a great deal of intuitive sense if you consider composite area problems to inherently contain distracters. Second, although we did not find strong support for our contention that spatial parsing is difficult for students, we feel that training students to quickly identify important perceptual chunks can still have a positive impact on performance. If, for example, students were trained to look for a “T” or “L” structure and map the segments onto the base and height of a shape, students might be less prone to developing shallow knowledge about area.

Annotated bibliography

1. Bransford (2000). How people learn: brain, mind, experience, and school National Academy Press.
2. Gonzales, P., Guzman, J. C., Partelow, L., Pahlke, E., Jocelyn, L., Kastberg, D., & Williams, T. (2004). Highlights From the Trends in International Mathematics and Science Study: TIMSS 2003. Washington, DC: National Center for Education Statistics.
3. Heffernan, N. T. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension. In Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society,307-312. Hillsdale, NJ: Erlbaum.
4. Koedinger, K. R., & Anderson, J. R. (1990). Abstract planning and perceptual chunks: Elements of expertise in geometry. Cognitive Science, 14, 511-550.
5. Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.
6. Koedinger, K. R., & Cross, K. (2000). Making informed decisions in educational technology design: Toward meta-cognitive support in a cognitive tutor for geometry. In Proceedings of the Annual Meeting of the American Educational Research Association (AERA), New Orleans, LA.
7. Mullis, I. V. S., Martin, M. O., Gonzalez, E. J., & Chrostowski, S. J. (2004). Findings from IEA’s Trends in International Mathematics and Science Study at the fourth and eighth grades. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
8. Owen, E., & Sweller, J. (1985). What do students learn while solving mathematics problems? Journal of Educational Psychology, 77, 272-284.
9. Simon, H. A., & Lea, G. (1974). Problem solving and rule induction: A unified view. In L. W. Gregg (Ed.), Knowledge and cognition. Hillsdale, NJ: Erlbaum.# Wilson, L. D. & Blank, R. K. (1999). Improving mathematics education using results from NAEP and TIMSS. Washington, DC: Council of Chief State School Officers, State Education Assessment Center.