PSLC Year 5 Projects

From LearnLab
Revision as of 13:40, 23 May 2009 by Koedinger (talk | contribs) (→‎Metacognition)
Jump to: navigation, search

New Year 5 projects

Refinement & Fluency CLUSTER ==> Cognitive Factors THRUST [Chuck]

Coordinative Learning CLUSTER ==> CF or Metacognition & Motivation THRUST [Ken]

Metacognition
Example-Rule Coordination
  • Roll- Labgebra - Inventing rules as preparation for future learning. Highlights that will go into it: 1) Last year we completed a study with 7 classes at Steel Valley Middle School. We got positive results - cognitive and motivational benefits. There is also a cogsci paper, which will be the basis for the updated Wiki page. 2) Over the year since then we built a tutoring system for IPL. 3) 10 days ago I finished another study in Steel Valley Middle School evaluating the tutor.
  • The Help Tutor Roll Aleven McLaren
  • **New** Aleven - Geometry_Greatest_Hits
Visual-Verbal Coordination
Motivation

Integrative Communication CLUSTER ==> Social Communicative THRUST [Chuck]

Computational Modeling and Data Mining THRUST [Ken]

Knowledge Analysis: Developing Cognitive Models of Domain-Specific Content
Learning Analysis: Developing Models of Domain-General Learning and Motivational Processes
Instructional Analysis: Developing Predictive Engineering Models to Inform Instructional Event Design

Notes

New thrusts "absorb" work from past clusters.

Integrated Thrust Summaries

Metacognition & Motivation Thrust

The work in this thrust builds on prior work started before the renewal, particularly work in the Coordinative Learning Cluster.

Metacognition

Past work within the Coordinative Learning Cluster emphasized to broad themes: Example-Rule Coordination and Visual-Verbal Coordination. These themes involve instruction that provides students with multiple input sources and/or prompts for multiple lines of reasoning. A good self-regulated learned needs to have the metacognitive strategies to coordinate information coming from different sources and lines of reasoning. We summarize Year 5 project results within these two themes as they address both whether providing multiple sources or reasoning prompts enhances student learning and whether metacognitive coordination processes can be supported or improved.

Example-Rule Coordination

Much of academic learning, particularly in Science, Math, Engineering, and Technology (SMET) domains but also in language learning, involves the acquisition of concepts and skills that must generalize across many situations if robust learning is to achieved. Often instruction expresses such generalizations explicitly to students with verbal descriptions, which we call "rules" (see the top-left cell in Figure XX). It may also communicate these generalizations by providing examples (bottom-left cell). Because "learning by doing" is recognized as critical to concept and skill acquisition, typical instruction also includes opportunities for students to practice application of the rules in "problems" (bottom-right cell). All to rarely, students are asked to generate rules themselves from examples of worked out problem solutions -- prompting students to do so is called "self-explanation" (top-right cell). The optimal combination of these four kinds of instruction (or instructional events) has been the focus on many projects that cut across math, science, and language domains. While typical instruction tends to focus on rules and practice opportunities (the main diagonal in Figure XX), these studies have now consistently demonstrated that a more balanced approach that includes at least as many examples and self-explanation opportunities leads to more robust learning.

PSLC studies in math, science, and language learning domains have been exploring the combination of worked-examples and self-explanation with computer-based tutoring during problem-solving practice. These studies bring together different research traditions 1) studies worked examples and cognitive load theory from Educational Psychology, particularly in Europe, 2) self-explanation studies primarily from cognitive science and psychology, and 3) intelligent tutoring system primarily from Computer Science.

As discussed in Schwonke, Renkel, Krieg, Wittwer, Aleven, & Salden (2009), past studies of worked example effects had compared against a control condition involving unsupported problem solving. This award-winning project has demonstrated the benefit of adding worked examples even in the context of a stronger control condition, namely, problem solving with instructional support of an intelligent tutor. Students spend take 20% less time in the example condition and learn as much or more on a variety of robust learning measures. The project has further demonstrated that a computer tutor that automatically adapts the transition from worked examples to problem solving leads to even further gains in robust student learning (Salden, Aleven, Renkl, & Schwonke, 2009).

Reflecting the benefits of a center in general and of the PSLC infrastructure in particular, this line of research has involved 5 laboratory studies and 3 in vivo studies run in labs and classrooms in Freiburg, Germany and Pittsburgh. These studies were all run in the context of the Geometry Cognitive Tutor, which automates delivery of complex instruction, insures reliable implementation of experimental differences, and provides rich process data (every 10 seconds) over hours of instruction. These studies involved more than 900 students and an average of 4 hours of learning time per student.

While the in vivo studies demonstrate that these substantial effects are robust to the high variability in real classroom studies, the associated lab studies allow more in depth investigation of learning process and learning theory. In particular, resent results from one of the lab studies reported in Schwonke et al. (2009) enhance theoretical understanding of complex human learning processes, particularly how and how deeply students choose to reason about instructional examples.

Table XX. Categories and examples of students' self-explanations

  • Principle-based explanation The learner verbalizes and elaborates on a mathematical principle. Mentioning a principle without some elaboration would not be coded as a principle-based explanation “Oh, that is major-minor arc, that means I’ve to subtract the minor arc from 360°”
  • Visual mapping The learner tries to relate content organized in different external representations and/or different visual tools (verbally as well as supported by gestures) “Where is angle ETF. Ah, has to be this one” (learner is pointing at corresponding spots in the graphic)
  • Goal–operator combination The learner verbalizes a (sub-)goal together with operators that help to accomplish this (sub-)goal “You can calculate this arc of a circle by subtracting 33° from 360°”

The example condition provided both more principle-based self-explanations and more visual mapping explanations whereas the problem condition provided many more goal–operator combination explanations. Principle-based and visual mapping self-explanations are consistent deeper processing that places more attention on the geometry rules and the non-trivial mapping of the rules to specific situations. These explanations suggest greater attention to the if-part or retrieval features of relevant knowledge components. Goal-operator explanations attend more to the arithmetic, that which must be done in problem-solving (the then-part). The arithmetic processing may be strengthening prerequisite knowledge, but is not directly relevant to the target geometric content. The greater number of principle-based and visual mapping self-explanations in the example group is consistent with the theory that example study frees cognitive resources so learners can engage in deeper processing. While problem-solving is beneficial later in learning (and thus the fading approach), in early learning it not only wastes time but it may put students in a performance-oriented mode (Dweck) whereby they do not as deeply process tutor instruction, which is equivalent to an example.

Investigations of correlations between self-explanation behavior further support this explanation, at least in part. Visual mapping explanations were significantly correlated with conceptual transfer, but principle-based explanations were not. Both principle-based and goal-operator explanations were significantly correlated with procedural transfer. Greater use of examples enhances deeper processing and that processing (at least of the mapping type) leads to greater conceptual understanding and transfer. Recall, that this achievement benefit was observed on top of substantial efficiency benefit in which example students needed 20% less instructional time.

  • Schwonke, R., Renkel, A., Krieg, C, Wittwer, J., Aleven, V., Salden, R. J. C. M. (2009). The Worked-example Effect: Not an Artefact of Lousy Control Conditions. Computers in Human Behavior, 25, 258-266.
  • Salden, R. J. C. M., Aleven, V. A. W. M. M., Renkl, A., & Schwonke, R. (2009). Worked examples and tutored problem solving: Redundant or synergistic forms of support? Topics in Cognitive Science, 1, 203-213.


The Center-mode focus on the issues of worked examples and self-explanation has allowed for cross-domain investigations of how these principles may further enhance student learning beyond already effective intelligent tutoring systems. These studies have been performed in Chemistry, Algebra, Physics, and English as a Second Language. The Chemistry studies replicated the result from Geometry that replacing half of the problems in a tutoring system with worked examples leads to more efficient learning. Across three studies, one at the college level and two at the high school level, McLaren et al (2008) found students learned just as much but in about 20% less time. If scaled to a semester long course, students using an example-based approach would have more than 3 free weeks!

  • McLaren, B.M., Lim, S., & Koedinger, K.R. (2008). When and How Often Should Worked Examples be Given to Students? New Results and a Summary of the Current State of Research. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 2176-2181). Austin, TX: Cognitive Science Society.

Other PSLC studies have identified benefits of worked examples and self-explanation in Physics (Nokes ref) and Algebra (Anthony ref).

A study in English is exploring whether the benefits of worked examples and self-explanation extend from math and science domains to language learning. This study is summarized in the Cognitive Factors section below. Interestingly, while there is some evidence that self-explanation helps in early learning, it does not appear to have as strong a benefit overall. This provides an important theoretical puzzle: Under what conditions is prompted self-explanation a productive strategy? Ongoing theoretical and empirical work is investigating this question.

More Direct Support for MetaCognition

One of the main challenges of education is to help students reach meaningful and robust learning. The assistance dilemma raises the question of what form (and ‘amount’) of assistance are most effective with different learners in different stages of the learning process (Koedinger & Aleven, 2007). Instruction followed by practice is known to be very efficient for teaching novices (e.g., Koedinger, Anderson, Hadley & Mark, 1997); yet, students often acquire shallow procedural skills, and fail to acquire conceptual understanding (Aleven & Koedinger, 2002). This can be attributed, at least in part, to students using superficial features and not encoding the deep features of the domain (Chi, Feltovich & Glaser, 1981). One approach to getting students to attend and encode the deep features is to add an invention phase prior to instruction. Invention as preparation for leaning (IPL) was shown to help students better cope with novel situations that require learning (Schwartz & Martin, 2004; Sears, 2006). In this process students are presented with a dilemma in the form of contrasting cases, and attempt to invent a mathematical model to resolve this dilemma. For example, Figure 1 shows four possible pitching machines. Students are asked to invent a method that will allow them to pick the most reliable machine. The concept of contrasting cases comes from the perceptual learning literature, since these cases, when appropriately designed, emphasize differences in the deep structure of the examples (Gibson & Gibson, 1955). The invention process includes designing a model, applying it to the given set of contrasting cases, evaluating the result, and debugging the model. This iterative process is very similar to the debugging process as described by Klahr and Carver (1988; Figure 2). Unlike other inquiry-based manipulations (cf. Lehrer et al., 2001; de Jong & van Joolingen, 1998), the goal of the IPL process is not for students to discover the correct model, but to prepare them for subsequent instruction.

1) Last year we completed a study with 7 classes at Steel Valley Middle School. We got positive results - cognitive and motivational benefits. There is also a cogsci paper, which will be the basis for the updated Wiki page. Domain knowledge:

  • IPL students in advanced classes were more capable of solving new strategy items without learning resource. In fact, in the absence of a learning resource, direct instruction students performed at floor, while IPL students performed as well as with the source.
  • This effect holds when controlling for simple domain knowledge (performance on normal items in the same test).
  • This was found in multiple new-strategy items. However, all results were found on a single topic (central tendency and graphing). The single test item on the topic of variability failed to capture difference between the groups.

Motivation: [Put below in motivation section or use as bridge to that section]

  • IPL students reported to have benefited more (F=3.3, p<.07)
  • There was a significant interaction between condition and test anxiety. Text anxiety was assessed using the MSLQ (Pintrich 1999) before the study began. Students who reported to have lower test anxiety also reported to have benefited more from IPL instruction compared to high-anxiety students in the no desin condition.
  • IPL students stayed more often in class to work during breaks (IPL: 16% No Design: 3%).
  • Furthermore, they did so during invention activities and not show-and-practice activities, suggesting that it is the activities that are motivating.

2) Over the year since then we built a tutoring system for IPL. The Invention Lab is an intelligent tutoring system for IPL. To give intelligent feedback, it uses two models:

  • A meta-cognitive model of the invention process
  • A cognitive model of the main concepts in the domain

This project we began at the beginning of the center to explore whether the benefits of tutoring could be achieved at the meta-cognitive level. Recent analysis of log files have revealed an exciting new finding. Past analysis had shown that the Help Tutor reduces students help-seeking errors while it was in place and giving immediate feedback, but we wanted to explore whether it would have a lasting impact and reduce such such errors later in the course. The second of two in vivo studies involved two different units of the Geometry Cognitive Tutor. The two units of study 2 were spread apart by one month. We collected data from students' behavior in the months between the two units and following the study. During these months students repeated previous material in preparation for statewide exams. As seen in the table below, the main effects of the help-seeking environment persisted even once students moved on to working with the native Cognitive Tutor! Overall, students who received help-seeking support during the study took more time for their actions following the study, especially for reading hints - their hint reading time before asking for additional hint is longer by almost one effect size in the month following the study (12 vs. 8 seconds, t(44)=3.0, p<.01). Also, students who were in the Help condition did not drill down through the hints as often (average hint level: 2.2 vs. 2.6, t(51)=1.9, p=.06). These effects are more consistently significant after both units, suggesting that having the study stretched across two units provided enough time for students to better acquire domain-independent help-seeking skills.



Visual-Verbal Coordination

Overall, student progress was slower than anticipated by the experimenters or the classroom teacher. Of the 83 students working in the intelligent tutor, 31 students (11 Control, 10 Visual Highlighting, 10 Visual Cueing) reached the last instructional unit (unit 3) during the experiment. For these students, results show that benefits of visual self-explanation for problem solving change over the course of tutoring practice (see the figure below). In the first instructional unit, students provided with visual cueing by the tutor are most accurate in their problem solving answers (M = .89, SD = .05) compared to students in the control condition (M = .83, SD = .06) or the visual-explanation condition (M = .85, SD = .05). Results demonstrated an overall effect of condition (F (2, 27) = 4.01, p = .03); post-hoc Bonferroni comparisons demonstrated that visual cueing significantly outperformed the control condition (p = .03) but not the visual self-explanation condition (p = .15), which fell between the two other groups. In contrast, by unit 3, students who visually self-explained the geometry principles (M = .86, SD = .08) were most accurate in their problem-solving answers, followed by the visual cueing condition (M = .83, SD = .11), and then the control condition (M = .73, p = .10). Results again demonstrated an overall effect of condition (F(2, 27) = 4.84, p = .016); post-hoc Bonferroni comparisons showed that the control condition was outperfomed by the visual self-explanations (p = .03) and the visual cueing (p = .05) conditions.

We analyzed overall posttest and delayed posttest results for students who had also taken the pretest. Posttest results demonstrated an overall improvement from pre- to posttest (F(1, 65) = 9.68, p = .03), but no significant condition differences (F<1). At delayed posttest, result suggested a test time (pretest vs. delayed posttest) by condition interaction (F(2. 37) = 2.87, p = .07). At delayed posttest (see Figure 2), students in the visual self-explanation condition outperformed students from the visual cueing condition and the control (interactive diagram) condition.

Image:EarliGraph.jpg 

   * Butcher, K. R., & Aleven, V. A. (in press 2009). Visual self-explanation during intelligent tutoring? More than attentional focus? European Association for Research on Learning and Instruction, 13th Biennial Conference. August 25-29, 2009: Amsterdam, the Netherlands.
Motivation

Consistent with the goals of the new Metacognition and Motivation Thrust, which will officially begin in Year 6, past PSLC projects have been begun investigating motivational issues. We summarize results of projects

Bringing it Together: Exploring Effects of Combining Principles

(Perhaps this should be saved for a cross-thrust section as there is CF, CMDM, and M&M involved.)

The main idea in the current project is to combine instructional interventions derived from four instructional principles. Each of these interventions has been shown to be effective in separate (PSLC) studies, and can be expected on theoretical grounds to be synergistic (or complementary). We hypothesize that instruction that simultaneously implements several principles will be dramatically more effective than instruction that does not implement any of the targeted principles (e.g. current common practice), especially if the principles are tied to different learning mechanisms. This project will test this hypothesis, focusing on the following four principles: 

   * Visual-verbal integration principle
   * Worked example principle
   * Prompted self-explanation principle
   * Accurate knowledge estimates principle

Building on our prior work that tested these principles individually, we have created a new version of the Geometry Cognitive Tutor that implements these four principles. We have conducted an in vivo experiment, and will conduct a lab experiment, to test the hypothesis that the combination of these principles produces a large effect size compared to the standard Cognitive Tutor, which does not support these principles or supports them less strongly. Analysis of the in vivo experiment is in progress.

Retrieved from "https://learnlab.org/wiki/index.php?title=PSLC_Year_5_Projects&oldid=9532"