Ringenberg Ill-Defined Physics

Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving?

Michael Ringenberg and Kurt VanLehn

Summary Table

PIs	Michael Ringenberg (Pitt) & Kurt VanLehn (Pitt)
LearnLab Course	Physics
Number of Students	N = 40

Abstract

Students who complete an introductory physics course often do not have a good conceptual understanding of the principles taught. There have been various attempts at increasing conceptual learning, often with only modest improvements. One promising avenue is the use of ill-defined problems. However, it can be very difficult for students to solve these problems without proper support. If ill-defined problem solving can be supported using intelligent tutoring systems, then it will be possible to investigate the potential of ill-defined problems and their influence on conceptual learning.

Background and Significance

The goal of a physics course is not to teach how to solve highly constrained \physics problems" but to increase a student's understanding of the world around them and how it can be modeled using equations and formulas. Although students are often good at solving the highly constrained physics problems that are part of homework and exams, they often lack an understanding of the concepts and principles exemplified by those problems [Halloun and Hestenes, 1985].

One possible reason for this is that students develop adequate strategies for solving physics problems, but these strategies are highly procedural and do not rely on a conceptual understanding of the material. There is evidence that students employ shallow problem solving methods. For example, when asked to sort problems, novices will rely on surface features of the problem like what objects (inclines, boxes, pucks, ice, etc.) are used and what quantities are specified (mass, velocity, coefficient of friction, etc.), whereas experts will match problems based on what principles are used to solve them [Chi et al., 1981].

More evidence for shallow problem solving methods includes observed use of examples. When referring back to examples or previously solved problems, novices will focus on matching surface features and try to mimic all the steps presented in the example [VanLehn and Jones, 1993]. However, when supplied with a relevant example that matches the current problem based on the principles used and may or may not match surface features, students tend to rapidly develop better principle-based problem analysis strategies [Ringenberg and VanLehn, 2006].

On the other hand, physics experts tend to solve these (and real-world) problems by first conceptualizing the problem, then forming a qualitative solution, and finally using relevant given quantities to derive a numeric solution [Larkin and Reif, 1979]. When students are required to perform more expert-like problem solving strategies, they can develop better conceptual knowledge. For example, when students are given the task of specifying which physics principles are needed to solve well-defined physics problems as part of their homework and exams (i.e. conceptualizing the problem), their understanding of these concepts improves more than just solving the problems [Leonard et al., 1996].

Unfortunately, simply requiring students to perform more expert-like problem solving strategies may notbe enough. The Andes tutoring system requires students to draw diagrams and specify necessary variables which are steps that experts perform, and students do better on these skills but do not perform any better on "conceptual" skills like principle identification [VanLehn et al., 2005]. A shortcoming of just prescribing and enforcing a problem solving strategy is that students may not see the value of using it and just adapt their shallow methods to include these extra steps.

Having physics students solve more ill-defined problems that are difficult to solve with shallow strategies seems like a viable method for encouraging more conceptual methods. According to [Simon, 1973], "ill-defined" problems covers a broad category of problem types, including problems that are missing necessary information, have multiple equally valid though not equivalent solutions, or have no definitive solution that experts can agree upon. Whereas experts often deal with these sorts of problems, students often have great difficulty solving these problems.

Programs like Activity-Based Physics and Project SCALE-UP break students up into peer groups to solve ill-defined problems. It is hard, however, to determine the effectiveness of the task in such situations as factors such as group size, variations in member's abilities, member's personalities, and gender composition all influence performance measures [Heller and Hollabaugh, 1992]. Maintaining an effective and productive group is very difficult, particularly when a single "bad apple" can negatively affect the entire group's performance [Felps et al., 2006].

Upon analysis of groups that successfully solve ill-defined problems, it was found that these groups tended to be more consistent at following a prescribed problem-solving strategy and are more likely to state the physics concepts and principles being used [Heller et al., 1992]. It should be possible to develop a system that can support solving ill-defined problems by enforcing the use of some better problem-solving strategies.

In the proposed study, participants will be presented with problems that lack some of the contextual clues of standard physics textbook problems. They will not reference idealized objects like "box A slides down an incline." Problems will cover concepts from Newton's Laws, Momentum, or Energy, which should prevent simple guessing on the part of the participants based on what they have studied or used most recently. In addition, participants in the "ill-defined"(experimental) condition will initially have all of the given quantities (mass, velocities, etc.) removed. This causes the problems to become ill-defined in that information necessary for producing a solution is missing. Participants in this condition will be asked to specify what information they would need to solve the problem. After they have done this, they will be provided with the necessary quantities and asked to solve the problem normally. This task is designed to get them to think about what principles they would used to solve the problem and how they are going to apply them. The hints and feedback they receive will facilitate this by asking what principles they plan to use and what would be necessary to use a given principle.

It is expected that participants in the ill-defined condition will develop more expert-like conceptual analysis as their shallow methods are not applicable for this task. This more expert-like conceptual analysis should be easily discernible using a [Dufresne et al., 1992] like problem matching task. In this task, participants are presented with a model problem statement and then asked which of two target problem statements would be solved most similarly to the model. Only one of the targets will use the same principles to solve it, but each of them may share surface features with the model. If participants are doing a "deeper" analysis, then they will select the targets that match based on the principles used to solve it, regardless of the amount of surface similarity. It is expected that engaging the principles in a more conceptual way will lead to greater conceptual understanding, as opposed to just thinking of them as labels for equations. To see if this task encourages a greater conceptual understanding, a multiple-choice conceptual inventory [Singh and Rosengrant, 2003] will be used to assess any gains made over the intervention.

Research question

Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving?

Independent variables

• Type of problems solved: ill-defined problems vs. well-defined problems

For this study, the ill-defined problems used lacked key information needed to solve the problem. The problem statements did not include the quantities needed to derive a numeric solution to the problem. Part of the task of solving these problems was to have the participants request the necessary quantities from the system. The system will provide hints and corrective feedback for this task. Once all of the necessary values are elicited by the participant, then the problem becomes a well-defined problem. The well-defined problems used were identical to the ill-defined problems except that all of the necessary information was given as part of the problem statement.

Figure 1. Example of one of the ill-defined problems. It is missing key information which would be required to produce a quantitative solution.

Regina is practising skateboard tricks. She grinds her board along a horizontal rail and falls of the end onto a mattress she placed there. How fast is she travelling just before hitting the mattress?

Figure 2. Example of the corresponding well-defined version of the problem in Figure 1. It includes all of the necessary and sufficient information needed to produce a quantitative solution to the problem.

Regina is practising skateboard tricks. She grinds her board along a horizontal rail and falls of the end onto a mattress she placed there. How fast is she travelling just before hitting the mattress? Height of rail: 0.5 m Regina's velocity when she leaves the rail: 0.45 m/s

Hypothesis

If students are required to figure out what information is needed to solve ill-defined physics problems before solving them, then they will develop better conceptual understanding than if they had been presented with the same problems with all the necessary information provided.

Dependent variables

• Normal post-test
  • Multiple-choice conceptual questions
• Transfer
  • Judgement task
    • Problem matching task: participants are given a target problem statement and are asked which of two additional problem statements are solved most similarly to the target problem without solving any of the problems (Dufresne, Gerace, Hardiamnn, & Mestre, 1992).
• Performance on Andes problems
  • Solution times
  • Error rates
  • Help requests

Expected Findings

Participants who solve the ill-defined versions of the problems will:

• Perform better on conceptual questions.
• Perform better on the problem matching task.
• Have faster solution times.
• Have lower error rates.
• Have fewer help requests.

Explanation

Because the experimental participants will be required to engage in more conceptual analysis of the problems, they will more deeply analyze and encode the knowledge components used in the problems. This will lead to better performance on tasks that use this better encoding. It will also have effects on problem solving because with the conceptual analysis done before problem solving, there will be less floundering and help abuse during problem solving.

Further Information

Annotated bibliography

• Paper and poster presented at ITS 2008 Conference (young researcher's track).

References

• [Chi et al., 1981] Chi, M. T. H., Feltovich, P., and Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5:121-152.

• [Dufresne et al., 1992] Dufresne, R. J., Gerace, W. J., Hardiman, P. T., and Mestre, J. P. (1992). Con- straining novices to perform expertlike problem analyses: Effects on schema acquisition. Journal of the Learning Sciences, 2(3):307-331.

• [Felps et al., 2006] Felps, W., Mitchell, T. R., and Byington, E. (2006). How, when, and why bad apples spoil the barrel: Negative group members and dysfunctional groups. Research in Organizational Behavior, 27:175-222.

• [Halloun and Hestenes, 1985] Halloun, I. A. and Hestenes, D. (1985). The initial knowledge state of college physics students. American Journal of Physics, 53(11):1043-1055.

• [Heller and Hollabaugh, 1992] Heller, P. and Hollabaugh, M. (1992). Teaching problem solving through cooperative grouping. part 2: Designing problems and structuring groups. American Journal of Physics, 60(7):637-644.

• [Heller et al., 1992] Heller, P., Keith, R., and Anderson, S. (1992). Teaching problem solving through cooperative grouping. part 1: Group versus individual problem solving. American Journal of Physics, 60(7):627-636.

• [Larkin and Reif, 1979] Larkin, J. H. and Reif, F. (1979). Understanding and teaching problem-solving in physics. International Journal of Science Education, 1(2):191-203.

• [Leonard et al., 1996] Leonard, W. J., Dufresne, R. J., and Mestre, J. P. (1996). Using qualitative problem- solving strategies to highlight the role of conceptual knowledge in solving problems. American Journal of Physics, 64(12):1495-1503.

• [Ringenberg and VanLehn, 2006] Ringenberg, M. A. and VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In Ikeda, M., Ashley, K., and Chan, T.- W., editors, ITS 2006, LNCS 4053, pages 625-634, Taiwan. Springer-Verlag Berlin Heidelberg. Winner of Best Paper First Authored by a Student Award.

• [Simon, 1973] Simon, H. A. (1973). The structure of ill-structured problems. Artificial Intelligence, 4(3):181- 201.

• [Singh and Rosengrant, 2003] Singh, C. and Rosengrant, D. (2003). Multiple-choice test of energy and momentum concepts. American Journal of Physics, 71(6):607-617.

• [VanLehn and Jones, 1993] VanLehn, K. and Jones, R. M. (1993). Better learners use analogical problem solving sparingly. In Utgoff, P. E., editor, Proceedings of the Tenth International Conference on Machine Learning, pages 338-345, San Mateo, CA. Morgan Kaufmann.

• [VanLehn et al., 2005] VanLehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. (2005). Andes physics tutoring system: Five years of evaluations. In McCalla, G., Looi, C. K., Bredeweg, B., and Breuker, J., editors, Artificial Intelligence in Education, pages 678-685, Amsterdam, Netherlands. IOS Pres.