Theory Wiki - User contributions [en]

Harnessing what you know

2009-05-14T17:32:58Z

Bobhaus:

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract'''. Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ''far transfer'', is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ''learning'' at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ''knowledge components''. This assumption has been formalized in computational models of human cognition, including ''production rules'' in the ACT-R architecture (Anderson & Lebiere, 1998) and ''chunks'' in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual's performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ''learning curves'', and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ''in vivo'' experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (''F'') that a charged particle (q) experiences when it is located in a region with an electric field (''E''). The relationship between these three quantities is summarized by the following equation: ''F = E*q''. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems.
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component's generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ''et al.'' analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ''c'', was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (\omega) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw's linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (\alpha) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (\theta) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ''distance'' for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ''meters''. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ''radians''. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 - 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject's factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (\alpha = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ''F''(2, 1641) = 3.33, ''p'' < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (''p'' = .01), but not rotational dynamics (''p'' = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ''F''(26, 4352) = 24.56, ''p'' < .001. The overall effect was qualified by a three-way interaction, ''F''(8, 4352) = 2.82, ''p'' = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (''p'' < .001) and dynamics (''p'' < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, "A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?" It would be tempting for a novice to match the word "angle" in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student's perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:25:05Z

Bobhaus: /* Harnessing what you know: The role of analogy in robust learning */

Harnessing what you know

2009-05-14T17:24:28Z

Bobhaus:

Harnessing what you know

2009-05-14T17:22:32Z

Bobhaus: /* The instructional unit as the knowledge component */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ''far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ''learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ''knowledge components’’. This assumption has been formalized in computational models of human cognition, including ''production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ''chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ''learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ''in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (''F’’) that a charged particle (q) experiences when it is located in a region with an electric field (''E’’). The relationship between these three quantities is summarized by the following equation: ''F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems.
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ''et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ''c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ''distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ''meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ''radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (‘‘n‘‘ = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level ( = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:22:03Z

Bobhaus:

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ''far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ''learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ''knowledge components’’. This assumption has been formalized in computational models of human cognition, including ''production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ''chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ''learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ''in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (''F’’) that a charged particle (q) experiences when it is located in a region with an electric field (''E’’). The relationship between these three quantities is summarized by the following equation: ''F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems.
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ''et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ''c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ''distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ''meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ''radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (‘‘n‘‘ = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level ( = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. The reason for including translational dynamics is to control for XXX? Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:18:23Z

Bobhaus: /* Discussion */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (a = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ''F''(2, 1641) = 3.33, ''p'' < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (''p'' = .01), but not rotational dynamics (''p'' = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ''F''(26, 4352) = 24.56, ''p'' < .001. The overall effect was qualified by a three-way interaction, ''F''(8, 4352) = 2.82, ''p'' = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (''p'' < .001) and dynamics (''p'' < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements. We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:18:05Z

Bobhaus: /* Discussion */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (a = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ''F''(2, 1641) = 3.33, ''p'' < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (''p'' = .01), but not rotational dynamics (''p'' = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ''F''(26, 4352) = 24.56, ''p'' < .001. The overall effect was qualified by a three-way interaction, ''F''(8, 4352) = 2.82, ''p'' = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (''p'' < .001) and dynamics (''p'' < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements. 
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:17:49Z

Bobhaus: /* Physics concepts as knowledge components */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (a = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ''F''(2, 1641) = 3.33, ''p'' < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (''p'' = .01), but not rotational dynamics (''p'' = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ''F''(26, 4352) = 24.56, ''p'' < .001. The overall effect was qualified by a three-way interaction, ''F''(8, 4352) = 2.82, ''p'' = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (''p'' < .001) and dynamics (''p'' < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:17:30Z

Bobhaus: /* The user-interface element as knowledge components */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (a = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ''F''(2, 1641) = 3.33, ''p'' < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (''p'' = .01), but not rotational dynamics (''p'' = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ''F''(26, 4352) = 24.56, ''p'' < .001. The overall effect was qualified by a three-way interaction, ''F''(8, 4352) = 2.82, ''p'' = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (''p'' < .001) and dynamics (''p'' < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:16:46Z

Bobhaus: /* The instructional unit as the knowledge component */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (a = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ''F''(2, 1641) = 3.33, ''p'' < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (''p'' = .01), but not rotational dynamics (''p'' = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:16:34Z

Bobhaus: /* The instructional unit as the knowledge component */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (a = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ''F''(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (''p'' = .01), but not rotational dynamics (''p'' = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:16:03Z

Bobhaus: /* The instructional unit as the knowledge component */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (a = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:15:47Z

Bobhaus: /* Data characteristics */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (''n'' = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level (a = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. The reason for including translational dynamics is to control for XXX? Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:15:08Z

Bobhaus: /* Introduction */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems. 
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (‘‘n‘‘ = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level ( = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. The reason for including translational dynamics is to control for XXX? Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:14:14Z

Bobhaus: /* Knowledge decomposition and learning curves */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems.
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (‘‘n‘‘ = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level ( = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. The reason for including translational dynamics is to control for XXX? Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:12:53Z

Bobhaus: /* Harnessing what you know: The role of analogy in robust learning */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''' Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems.
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (‘‘n‘‘ = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level ( = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. The reason for including translational dynamics is to control for XXX? Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

Harnessing what you know

2009-05-14T17:11:18Z

Bobhaus: /* Curriculum and knowledge-component mapping */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''''. Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems.
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.jpg|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (‘‘n‘‘ = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level ( = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. The reason for including translational dynamics is to control for XXX? Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

File:CONCEPTS.jpg

2009-05-14T17:10:43Z

Bobhaus:

File:INTERFACE.jpg

2009-05-14T17:10:32Z

Bobhaus:

File:KCUNIT.jpg

2009-05-14T17:10:22Z

Bobhaus:

File:UNITS.jpg

2009-05-14T17:10:09Z

Bobhaus:

File:DRAW-EFIELDS-VECTOR2.jpg

2009-05-14T17:09:35Z

Bobhaus:

Harnessing what you know

2009-05-14T17:09:25Z

Bobhaus: /* Harnessing what you know: The role of analogy in robust learning */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

'''Abstract''''. Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present paper is to reconcile these apparently incompatible beliefs. Toward that end, we will use a repository of data, taken from the Physics LearnLab, to argue that the level of granularity of the constituent knowledge components affects the detection of to transfer from one domain to another. 

==Introduction==
In well-structured domains, such as math or science, teachers often presume that the contents of one unit will transfer to units taught later in the semester; however, the learning literature is replete with evidence suggesting that transfer, especially ‘’far transfer’’, is difficult to achieve (Detterman, 1993). Do teachers have unrealistic expectations of their students, or are scientists looking in the wrong places to find evidence of far transfer? The primary goal of the present paper is to seek a resolution to this potential contradiction. Toward that end, we will define ‘’learning’’ at multiple levels of granularity and show how different levels of knowledge disaggregation reveal different conclusions about the existence or non-existence of far transfer. 

===Knowledge decomposition and learning curves===
Many domains, such as math, science, and computer programming, assume that knowledge can be decomposed into a partially ordered set of skills or ‘’knowledge components’’. This assumption has been formalized in computational models of human cognition, including ‘’production rules’’ in the ACT-R architecture (Anderson & Lebiere, 1998) and ‘’chunks’’ in the SOAR architecture (Newell, 1990). 
Evidence for the psychological plausibility for knowledge components can be found in the shape of the curve when an individual’s performance, which is typically measured as an error rate or the elapsed time, plotted against the opportunities to apply that particular piece of knowledge. These graphs are often referred to as ‘’learning curves’’, and an idealized learning curve monotonically decreases over time. Classic examples of learning curves include memorizing non-sense syllables (Ebbinghaus, 1913), learning how to roll a cigar (Crossman, 1959), and transmitting Morse code (Bryan & Harter, 1897). 
A more contemporary example of a learning curve can be found in the domain of electrodynamics (Hausmann & VanLehn, under review). Students enrolled in a second-semester physic course were asked to solve problems with the Andes Physics Tutor (VanLehn et al., 2005). During an ‘’in vivo’’ experiment (Hausmann & VanLehn, 2007), students were asked to solve four electrodynamics problems, which included calculating the magnitude of an electric force (‘’F’’) that a charged particle (q) experiences when it is located in a region with an electric field (‘’E’’). The relationship between these three quantities is summarized by the following equation: ‘’F = E ∙ q’’. Before students are allowed to write an equation in the Andes, however, they must first define all of their variables, which includes drawing an electric-field vector. The learning curve for the experiment can be found in Figure 1. 

[[Image:DRAW-EFIELD-VECTOR1.jpg|Figure 1. The learning curve for drawing an electric-field vector in Andes.]] 

Upon first glance, three features are immediately evident in Figure 1. First, the first opportunity to draw an electric-field vector is actually the lowest error rate of all five opportunities. This directly contradicts the power law theory of learning. Second, there is a steady progression from a relatively high error rate for the second opportunity to the last. This segment of the graph is aligned with our expectations. Finally, the description of this dataset intimated that there were only four opportunities because the problem set given to the students during the experiment only consisted of four problems. This particular learning curve plots five opportunities. This final data-point represents only two students, so that suggests that these two individuals drew an extra electric-field vector while solving one of the four problems.
How do we reconcile this particular learning curve with the predictions of many learning theories? One potentially useful solution is to reanalyze the knowledge component itself. Vectors represent both magnitude and direction. The direction of a vector in Andes is set in a dialog box in the interface. For the first opportunity, the problem statement gives the precise angle in which the students are supposed to draw the vector. For all of the other opportunities, the students are responsible for calculating or inferring the direction of the electric-field vector. From a task analysis, we could argue that drawing an electric-field vector, when it is given in the problem statement, is a separate knowledge component from inferring the direction of a vector. The shape of the learning curve in Figure 2 supports this hypothesis. 

[[Image:DRAW-EFIELDS-VECTOR2.jpg|Figure 2. A reanalysis of the electric-field vector decomposed into two new knowledge components]] 

The process or methodology used to reanalyze a knowledge component’s generality was based on (Corbett, McLaughlin, & Scarpinatto, 2000). When Corbert ‘’et al.’’ analyzed the learning curves of 34 students applying the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000, p. 101). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, ‘’c’’, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000, p. 102). 

===Curriculum and knowledge-component mapping===
In a typical introductory physics course, translational kinematics (e.g., equations describing the motion of a particle along a straight trajectory) is taught during the second week of the semester. Rotational kinematics (e.g., equations describing the motion of an extended body in a circular trajectory) is typically taught during the eighth week of the course. For the present purposes, the data used for our analyses were taken from students enrolled in the first semester of introductory physics at the US Naval Academy. A condensed version of their syllabus is listed in Table 1. 

[[Image:UNITS.pgn|Table 1. The sequence of General Physics I units taught at the US Naval Academy.]] 

The mapping between translational and rotational knowledge components is fairly straightforward for most knowledge components. There are, however, some interesting differences. 

====Linear (v) vs. Angular (ω) Velocity====
The analogy between linear (i.e., translational) and angular (i.e., rotational) velocity is a straightforward mapping due to a special problem-solving heuristic. Angular velocity can be transformed into linear velocity by imagining the head of a screw that moves linearly as the rotating body turns. As the body turns, it unwinds the screw. The result is that the screw’s linear velocity is directly proportional to the angular velocity of the rotating body. If the conditions are set so that the threads on the screw are equal to one revolution of the body, then they can be placed in a 1:1 relationship. Given the translatability between the two, we predict positive transfer between linear and angular velocity. 

====Linear (a) vs. Angular (α) Acceleration====
The heuristic for relating linear to angular velocity also works for acceleration. As the extended body speeds up or slows down, so does the head of the imaginary screw. Because of the tight connection between the two units, we predict there will be positive transfer for linear and angular acceleration. 

====Linear (s) vs. Angular (θ) Displacement====
The same, however, is not true for linear and angular displacement. Instead of a one-to-one mapping between the two, a new concept needs to be learned. In the linear case of displacement (which first needs to be distinguished between ‘’distance’’ for many students), the displacement is a resultant vector that points from the beginning of the interval of interest to the end of the interval. The displacement of a particle can be imagined as a straight line, and it is measured in ‘’meters’’. Most students have a vast amount of experience by the time they take physics I. Angular displacement, on the other hand, is a measure of the angle through which an extended body turns over an interval of time, and it is measured in ‘’radians’’. Individuals typically do not have as much experience talking or thinking about movement as a change in angle. Therefore, we would not predict transfer in the case of displacement because angular displacement is a new idea that does not have as strong of a basis in everyday interactions with the physical world. 

==Analyses and Results==
===Data characteristics===
The data analyzed for this project were taken from three semesters (Fall 2005 – 07) of college physics taught at the United States Naval Academy (USNA). Most students were sophomores, and they used the Andes Physics Tutor to solve their homework assignments. The data were downloaded from a central data repository called the DataShop, which is hosted by the Pittsburgh Science of Learning Center. For the analyses reported below (i.e., translational kinematics, translational dynamics, and rotational kinematics), the sample size consisted of two-hundred and twenty-one students (‘‘n‘‘ = 221) who generate 76,891 transactions. 
Our analyses are structured as follows. First, we conducted an ANOVA for each knowledge component model, testing for differences between units. We also used opportunity as a within subject’s factor. To explore differences within each opportunity, we conducted pair-wise comparisons between units for each opportunity. Because of the large sample size, we adopted a conservative alpha level ( = .01). Finally, we restricted our analyses to the first three opportunities because the number of observations drops precipitously for each successive opportunity. 

===The instructional unit as the knowledge component===
The first knowledge component treated each unit as a separate knowledge component. Because we were initially interested in far transfer, we included two units: translational and rotational kinematics. We also included a third unit, translational dynamics, as a control case. The reason for including translational dynamics is to control for XXX? Translational dynamics occurred after translational kinematics, but before rotational kinematics. Therefore, we would expect the learning curves for translational dynamics to fall somewhere between translational and rotational kinematics. The learning curves, over three opportunities, can be found in Figure 3. 

[[Image:KCUNIT.jpg|Figure 3. A using the entire unit as a single knowledge component.]] 

For the first opportunity, there was a statistically reliable difference between the three units, ‘‘F‘‘(2, 1641) = 3.33, ‘‘p‘‘ < .001. Translational kinematics was the easiest of the three units because it had the lowest assistance score for the first opportunity. It demonstrated a reliably lower assistance score than rotational kinematics (‘‘p‘‘ = .01), but not rotational dynamics (‘‘p‘‘ = .35). There were no differences between the three units for the second and third opportunities. 

===The user-interface element as knowledge components===
The overall shape of the learning curves for the three units were roughly monotonic, there was one problem. The theory of transfer would predict that rotational dynamics and rotational kinematics would demonstrate lower assistance scores because they came later in the semester. Therefore, we decided to break down these broad knowledge components into knowledge components related to the Andes user interface: drawing vectors, defining scalar quantities, and writing equations. The learning curves associated with these knowledge components can be found in Figure 4. 

[[Image:INTERFACE.jpg|Figure 4. A decomposition of each unit into knowledge components that correspond to the user interface.]] 

Overall, there was a reliable difference between units, opportunities, and knowledge components, ‘‘F‘‘(26, 4352) = 24.56, ‘‘p‘‘ < .001. The overall effect was qualified by a three-way interaction, ‘‘F‘‘(8, 4352) = 2.82, ‘‘p‘‘ = .004. Using Figure 4 as a guide, we restricted our analyses to just the vector knowledge components as the students progressed through the curriculum. It appears that the amount of assistance needed to correctly apply a vector knowledge component grew with time. For the first opportunity, more assistance was needed to draw vectors in rotational kinematics than in the case of translational kinematics (‘‘p‘‘ < .001) and dynamics (‘‘p‘‘ < .001). The shape of the curves for the other two knowledge components was reasonable for the first opportunity. 

===Physics concepts as knowledge components===
The analyses from the previous section suggest a closer examination of the vector learning curves. As the students move through the semester, they demonstrated slowly escalating assistance scores for drawing vectors. This is a very clear case where transfer is not occurring. Therefore, we decided to break down the vectors into their constituent physical concepts, which included drawing the acceleration, velocity, and displacement. The decomposed vector knowledge components are shown in Figure 5. 
According to the learning curves, it appears there is no transfer between drawing a translational displacement vector and drawing an angular displacement vector. At least initially, there is a huge jump between the first opportunity to apply this particular knowledge component (DRAW-DISPLACEMENT & DRAW-ANG-DISPLACEMENT), and then the assistance score returns to a low, asymptotic level. 
One potential explanation for the initial increase in assistance scores for displacement is in the way most rotational kinematics problems are worded. For example, the first problem in the USNA rotational homework set is, “A wheel is rotating counterclockwise at a constant rate of 3 rotations per second. Through what angle does the wheel rotate in 60.0 s?” It would be tempting for a novice to match the word “angle” in the problem statement, and use that as a basis for defining an angle in the Andes user interface. However, once the student attempts to define an angle, then the tutor will provide an unsolicited error message indicating that the angle is not part of the solution path for this problem. If the student then draws a displacement vector, then all of the errors and hints are blamed on the DRAW-ANG-DISPLACEMENT knowledge component (i.e., we use a temporal heuristic for the assignment of blame problem, Nwaigwe, Koedinger, VanLehn, Hausmann, & Weinstein, 2007). 

[[Image:CONCEPTS.jpg|Figure 5. A decomposition of the user-interface vector knowledge components into the corresponding physical concepts.]] 

==Discussion==
In the introduction, we pointed out the observation that there is an apparent contradiction between the empirical results investigating far transfer and the assumptions that teachers make within their own classroom. Teachers expect that their students should retain the knowledge components over several weeks, often with many other intervening units of instruction. However, the learning literature on far transfer seems to suggest that it is a rare occasion when knowledge lasts over long retention intervals. 
To resolve the discrepancy between theory and practice, we introduced the hypothesis that the granularity of the assessed knowledge plays a large role in whether transfer is observed or not. For example, when the unit was taken as the knowledge component, then there was absolutely no evidence of transfer. The assistance scores associated with translational kinematics was initially lower (i.e., the first opportunity) than both the translational dynamics and rotational kinematics units. This initial advantage was maintained over fourteen of the sixteen opportunities. 
Because there was no evidence of any sort of transfer, we decomposed the large, unit-size knowledge components into three smaller knowledge components that corresponded to the three broad categories of user-interface elements.
We repeated this process for the user interface elements that were vectors because the learning curves suggested that there was a drift toward increasing assistance score values. For the most part, the equations and scalar definitions were decreasing as the semester advanced. The vectors were disaggregated into acceleration, velocity, and displacement. These categories were more sensible because they finally corresponded to the concepts that are taught in the physics textbook. 
Future work will include better understanding why the displacement vector showed such a steep learning curve. At first, students were asking for lots of help and committing many mistakes. However, after making those initial attempts, they seemed to learn how to apply this knowledge component fairly quickly. We also plan to extend our analyses to include the equations that were written. From the student’s perspective, writing equations is the most important part of the course. 

==References==
# Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Mahwah, N.J.: Lawrence Erlbaum Associates.
# Bryan, W. L., & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27-53.
# Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81-108.
# Crossman, E. (1959). A theory of acquisition of speed-skill. Ergonomics, 2(2), 153-166.
# Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 1-24). Norwood, NJ: Ablex.
# Ebbinghaus, H. (1913). Memory. A Contribution to Experimental Psychology. New York: Teachers College, Columbia University.
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press.
# Hausmann, R. G. M., & VanLehn, K. (under review). The effect of generation on robust learning. International Journal of Artificial Intelligence and Education.
# Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
# Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R. G. M., & Weinstein, A. (2007). Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (pp. 246-253). Amsterdam: IOS Press.
# VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.

File:DRAW-EFIELD-VECTOR1.jpg

2009-05-14T17:09:05Z

Bobhaus:

Prompted self-explanation hypothesis

2009-01-23T14:50:30Z

Bobhaus: Redirecting to Prompted self-explanation principle

#REDIRECT [[Prompted self-explanation principle]]

== Brief statement of principle ==
When students are given a [[worked examples|worked example]] or text to study, prompting them to self-explain each step of the worked example or each line of the text causes higher learning gains than having them study the material without such prompting.

== Description of principle ==
Many empirical studies have shown that there is a large amount of variance when it comes to individually produced [[Self-explanation|self-explanations]] (Chi et al., 1989). Some students have a natural tenancy to self-explain, while other students do little more than repeat the content of the example or expository text. The quality of the self-explanations themselves can be highly variable (Lovett, 1992; Renkl, 1997). One instructional intervention that has been shown to be effective is to prompt students to self-explain (Chi et al., 1994). [[Prompting]] can take many forms, including verbal prompts from human experimenters (Chi et al., 1994), prompts automatically generated by computer tutors (McNamara, 2004; Hausmann & Chi, 2002; Aleven & Koedinger, 2002), or embedded in the learning materials themselves (Hausmann & VanLehn, 2007).

In the context of studying an example or reading a text, prompting for [[Self-explanation|self-explanations]] leads to greater learning gains than naturally occuring student practices.

=== Operational definition ===
* Self-explaining is defined as a "content-relevant articulation uttered by the student after reading a line of text" (Chi, 2000; p. 165) or after studying a step in a worked-out example. A self-explanation may contain a meta-cognitive statement and/or a self-explanation inference.
** A meta-cognitive statement is an assessment, made by the student, of his or her own current understanding of the line of text or example step.
** A self-explanation inference is "an identified pieced of knowledge generated...that states something beyond what the sentence explicitly said" (Chi, 2000; p. 165).
*Prompting is defined as an external cue that is intended to elicit the activity of self-explaining. Prompts are typically generated by a person, tutoring system, or a verbal reminder embedded in the learning material.

=== Examples ===
Here are the instructions to self-explain, taken from Chi et al. (1994):

"We would like you to read each sentence out loud and then explain what it means to you. That is, what 
new information does each line provide for you, how does it relate to what you've already read, does it give 
you a new insight into your understanding of how the circulatory system works, or does it raise a question 
in your mind. Tell us whatever is going through your mind–even if it seems unimportant." 

These prompts were reworded to be used in Hausmann & VanLehn (2007):

* What new information does each step provide for you?
* How does it relate to what you've already seen?
* Does it give you a new insight into your understanding of how to solve the problems?
* Does it raise a question in your mind?

These prompts were then included as text, just below a worked-out example. The example was presented as a video taken of the Andes interface, with a voice-over narration describing the user-interface actions (see Table below). In this example, the student is learning how to solve the following problem:

<Blockquote>A charged particle is in a region where there is an electric field E of magnitude 
14.3 V/m at an angle of 22 degrees above the positive x-axis. If the charge on the particle 
is -7.9 C, find the magnitude of the force on the particle P due to the electric field E.</Blockquote>

 

{| cellspacing="0" cellpadding="5" border="1"
|+ '''An example of prompting for self-explanining'''
|-
| style="border-bottom: 3px solid grey;" |
    Now that all the given information has been entered, we need to apply our knowledge of physics to solve the problem. 

    One way to start is to ask ourselves, “What quantity is the problem seeking?” In this case, the answer is the magnitude of the force on the particle due to the electric field. 

    We know that there is an electric field. If there is an electric field, and there is a charged particle located in that region, then we can infer that there is an electric force on the particle. The direction of the electric force is in the opposite direction as the electric field because the charge on the particle is negative.

    We use the Force tool from the vector tool bar to draw the electric force. This brings up a dialog box. The force is on the particle and it is due to some unspecified source. We do know, however, that the type of force is electric, so we choose “electric” from the pull-down menu. For the orientation, we need to add 180 degrees to 22 degrees to get a force that is in a direction that is opposite of the direction of the electric field. Therefore we put 202 degrees. Finally, we use “Fe” to designate this as an electric force.

<center>[ PROMPT ]</center>

    Now that the direction of the electric force has been indicated, we can work on finding the magnitude. We must choose a principle that relates the magnitude of the electric force to the strength of the electric field, and the charge on the particle. The definition of an electric field is only equation that relates these three variables. We write this equation, in the equation window.

<center>[ PROMPT ]</center>

|}
Note. PROMPT = "Please begin your self-explanation."

== Experimental support ==

=== Laboratory experiment support ===
Prompting for self-explaining has been shown to be effective in both increasing the amount, as well as learning gains (Chi et al., 1994). Prompting for self-explaining is typically paired with a training session, which instructs students on how to produce explanations. Laboratory research has shown that both the training and prompting techniques can be effective in producing performance gains (Bielaczyc, Pirolli, & Brown, 1995). Training does not necessarily have to be done by a human tutor. Instead, training students to self-explain can be automatized with a computerized training system (McNamara, 2004).

=== In vivo experiment support ===

Several in vivo experiments have leveraged laboratory work for inclusion of self-explaining in the classroom. Some in vivo experiments include:

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]
*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]
*[[Craig_questions|Deep-level questions during example studying (Craig, VanLehn, & Chi, 2006)]]
*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation|Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

== Theoretical rationale ==

Prompting for self-explaining should increase the probability that a student engages in self-explaining, which includes an increase in the amount and accuracy of meta-cognitive monitoring statements and self-explanation inferences. Prompting for self-explaining is an attempt to increase the likelihood of traversing deep learning events.

{| border="1" cellpadding="5" cellspacing="0"
|-
|Start
# Process the line shallowly, e.g., paraphrasing it 
## There is nothing more to learn => Exit, with learning 
## The line is incomplete; its explanation is missing => Exit, with little learning 
# Try to process the line deeply, e.g., self-explain it 
## There is nothing missing from the line => Exit, with learning 
## The line is incomplete; its explanation is missing 
### The attempted self-explanation succeeds => Exit, with learning 
### The attempted self-explanation fails => Exit, with perhaps less learning 
|-
|}

== Conditions of application ==

When should a prompt for self-explanation be delivered? In many of the studies described on this page, prompts for self-explanation were offered after each step of a worked-out solution. The timing of the prompt may depend on the domain. For example, in Hausmann and VanLehn (2007), the domain was physics, which requires the acquisition of procedure knowledge. The prompt to self-explain was issued after each solution step. For a more conceptual domain, such as the circulatory system, the experimenter in Chi et al. (1994) prompted the students to self-explain after reading each page of a text on the circulatory system. Roughly one line (or idea) was contained on each page of the text. After several pages, the participants became accustomed to the procedure, and turning the page became an implicit prompt for the students to begin self-explaining (Chi, personal communication).

== Caveats, limitations, open issues, or dissenting views ==
Examples typically precede problem solving. For example, in Sweller and Cooper (1985; Experiment 2), they asked students to study 2 examples in preparation to solve 8 problems. Similarly, Chi et al. (1989) asked students to read through 4 chapters of a physics text, which contained several examples. After studying each chapter, the students were asked to solve problems related to the content that they just studied. Finally, Trafton and Reiser (1993) manipulated the presentation of examples and problems by using either a blocked design, where students studied 6 examples, then solved 6 problems. Alternatively, an alternating conditions presented one example first, then solved one problem. They continued this sequence until all problems and examples were completed.

The order of solving and studying examples from Hausmann and VanLehn (2007) differed from traditional research on example-studying. In their experiment, students attempted to solve a problem first, and then studied an isomorphic example. The students alternated between solving problems and studying examples until all four problems were solved and all three examples were studied. Problems were presented first to capitalize on the strengths of impasse-driven learning (VanLehn , 1988). The problems created conditions where an impasse might be reached while solving a problem, and the example would demonstrate a smooth, expert solution to the same problem.

== Variations (descendants) ==
[[Corrective self-explanation]]

== Generalizations (ascendants) ==
[[Example-rule coordination principle]]

== References ==

Aleven, V. A. W. M. M., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explain with a computer-based Cognitive Tutor. Cognitive Science, 26, 147-179. [http://dx.doi.org/10.1016/S0364-0213%2802%2900061-7]

Bielaczyc, K., Pirolli, P., & Brown, A. L. (1995). Training in self-explanation and self-regulation strategies: Investigating the effects of knowledge acquisition activities on problem solving. Cognition and Instruction, 13(2), 221-252. [http://scholar.google.com/scholar?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=zUR&q=%22training+in+self-explanation+and+self-regulation+strategies:+Investigating+the+effects+of+knowledge+acquisition+activities+on+problem+solving%22&um=1&ie=UTF-8&sa=N&tab=ws]

Chi, M. T. H., DeLeeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, 439-477. [http://www.pitt.edu/~chi/papers/ChiBassokLewisReimannGlaser.pdf]

Hausmann, R. G. M., & Chi, M. T. H. (2002). Can a computer interface support self-explaining? Cognitive Technology, 7(1), 4-14. [http://www.pitt.edu/~bobhaus/hausmann2002.pdf]

Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press. [http://learnlab.org/uploads/mypslc/publications/hausmannvanlehn2007_final.pdf]

Lovett, M. C. (1992). Learning by problem solving versus by examples: The benefits of generating and receiving information. In Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society (pp. 956-961). Hillsdale, NJ: Erlbaum.

McNamara, D. S., Levinstein, I. B., & Boonthum, C. (2004). iSTART: Interactive strategy training for active reading and thinking. Behavioral Research Methods, Instruments, and Computers, 36, 222-233. [http://www.ingentaconnect.com/content/psocpubs/brm/2004/00000036/00000002/art00007]

Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. Cognitive Science, 21(1), 1-29. [http://dx.doi.org/10.1016/S0364-0213(99)80017-2]

Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59-89. [http://scholar.google.com/scholar?hl=en&lr=&client=firefox-a&cluster=16552570726007249431]

Trafton, J. G., & Reiser, B. J. (1993). The contributions of studying examples and solving problems to skill acquisition. In Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society (pp. 1017-1022). Hillsdale, NJ: Erlbaum. [http://citeseer.ist.psu.edu/rd/40331946%2C149956%2C1%2C0.25%2CDownload/http://citeseer.ist.psu.edu/cache/papers/cs/2910/http:zSzzSzwww.aic.nrl.navy.milzSz%7EtraftonzSzpaperszSzcogsci93-exp1.pdf/the-contributions-of-studying.pdf]

VanLehn, K. (1988). Toward a theory of impasse-driven learning. In H. Mandl & A. Lesgold (Eds.), Learning issues for intelligent tutoring systems (pp. 19-41). New York: Springer.

[[Category:Glossary]]
[[Category:Instructional Principle]]

Harnessing what you know

2008-10-20T14:04:42Z

Bobhaus: /* How can far transfer be supported? */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

=== Abstract ===
Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present project is twofold. First, we will use educational data-mining models to identify knowledge components from translational kinematics that fail to transfer to rotational kinematics. Second, we will design an intervention, based upon cognitive principles from self-explanation and analogical comparison, to support knowledge components that fail to transfer.

=== Background and Significance ===
Traditional pedagogy assumes knowledge transfers between problems, units, and even courses; however, the learning literature suggests transfer is rarely observed (Detterman, 1993). Is there transfer between units in a complex science course, such as physics? If so, to what extent?

==== Research Objectives ====
Phase 1. Revise the initial knowledge-component model from the Andes physics tutor for both the translational and rotational kinematics units. 
Phase 2. Develop educational data-mining models to detect the success and failure of the transfer of knowledge components. Student profiles will be defined in an effort to aggregate over individual differences in tutored help-seeking and problem-solving strategies, while still being sensitive to them. 
Phase 3. Design an instructional intervention, based on cognitive science principles, to facilitate transfer between units. The format of the intervention will be designed around the literature on analogical comparison and self-explanation. The content of the intervention will be based on the revised knowledge-component model, the identification of failed knowledge-component transfer, and student profiles. 

==== Hypotheses ====

H1: The learning curves from translational kinematics knowledge components can predict the error rates for rotational kinematics. 
H2: Educational interventions that draw upon prior knowledge, such as analogical comparison and self-explanation, can support knowledge components that fail to transfer between translational and rotational kinematics. 

=== Prior Work ===
==== Near and Far Transfer ====
Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that learning is cumulative and that advanced courses will build on the knowledge and skills acquired in the introductory and foundational courses. This is especially true for the STEM disciplines, where students take a highly structured sequence of courses. To illustrate this assumption, consider the case of introductory physics, which is typically split among two semesters. In a traditional curriculum, the first semester covers Mechanics, and the second semester covers Electricity and Magnetism. Instructors of Physics II assume that the material learned in the first semester is retained and can be applied to the problems related to the motion of a charged particle in an electric or magnetic field. Moreover, most pedagogy assumes within-class transfer as well. That is, topics and concepts taught later in a course build upon and extending those taught earlier. 
Unfortunately, research on human cognition has shown that knowledge transfer (especially far transfer to novel contexts and applications) is much more rare than traditional pedagogy assumes (for reviews, see Barnett & Ceci, 2002; Bransford, Brown, & Cocking, 2000). For example, in a classic study on transfer, Gick and Holyoak (1980) asked participants to solve a difficult insight problem (i.e., the solution rate was 8%). Before solving this difficult problem, all of the participants read a story that proposed an analogous solution. Half of the participants received a hint that the story will help with the solution, whereas the other half of the students did not receive a hint. The results were clear. The solution rate was much lower (i.e., 20%) for the participants who did not receive any hints, whereas those who received hints demonstrated a dramatic increase in their solution rate (i.e., 92%). These results suggest that spontaneous far transfer is difficult for students to implement. 
However, because a fifth of the students were able to spontaneously transfer their knowledge of one domain to another, Gick and Holyoak (1980) demonstrate that spontaneous far transfer is indeed possible. With the appropriate scaffolding in place, it becomes quite likely. This is also true for children learning authentic science material. For instance, Brown and Kane (1988) taught pre-school children animal defense mechanisms such as mimicry. The children’s ability to transfer the concept of mimicry to other animals depended crucially on their depth of understanding. That is, if the child understood mimicry at the level of the causal structure, then they were more likely to demonstrate transfer; whereas if the child was only imitating the behavior of the teacher, then they failed to transfer the concept. 

==== What transfers? ====
Often, the debate surrounding whether far transfer is tenable must address the issue of the unit of analysis. In other words, what transfers? Several hypotheses have been posited, including the doctrine of formal discipline from antiquity, Thorndike’s theory of identical elements, and Singley and Anderson’s (1989) identical-productions theory of transfer. The formal discipline theory implicated entire domains of knowledge were the units of analysis. For instance, politicians would be well advised to learn mathematics because it will cause them to be quicker thinkers (Lehman, Lempert, & Nisbett, 1988). In other words, the mind is analogous to a muscle that, when exercised properly, will increase in strength.
However, early psychologists took issue with the doctrine of formal discipline and challenged it on empirical grounds. Thorndike and Woodword (1901a; 1901b; 1901c) demonstrated, in an impressive series of studies, that transfer could only be expected if the two tasks shared “common elements.” For example, receiving training on estimating the area of a rectangle did not reduce the error rate of estimating the area of a different shape (e.g. triangle). 
Similar findings have been demonstrated with abstract reasoning tasks. For instance, Wason (1968) developed a deceptively simple task to assess an individual’s ability to reason about a bi-conditional rule. First-year psychology and statistics students were asked to evaluate the following rule: “If there is a D on one side of any card, then there is a 3 on its other side.” Then they were shown four cards that had a symbol on one side and another symbol in brackets indicating the contents of the back of the card. The cards were: D(3), 3(K), B(5), 7(D). The cards were placed in random order in front of the participant, and the experimenter pointed to each card and asked if that card could be used to determine if the rule was true or false. Collapsing across conditions, only 14.7% of the participants were able to correctly identify the cards that tested the veracity of the rule.
In a follow-up study, Cheng, Holyoak, Nisbett, and Oliver (1986) investigated the conditions under which formal training can enhance performance on abstract tasks, such as the Wason 4-card selection task. They found, after an entire semester of instruction on logic, there was no difference in the error rate on the Wason task (Exper. 2; p. 306). Even more to the point, Cheng et al. created their own training materials that were specifically designed to improve logical reasoning. Again, they found that performance on the Wason four-card task was not improved by their customized formal instruction alone (Exper. 1). 
From the available evidence, it appears that entire disciplines are not the unit of transfer, nor is the proposal of common elements of transfer specific enough to make predictions about what exactly transfers between two learning situations. A more specific theory of what constitutes an “element” is Singley and Anderson’s (1989) hypothesis that production rules, or skills, are the unit of transfer. In their analysis of learning how to use text editors, they demonstrated that the surface features can vary substantially, yet the production rules that compose the cognitive skill are transferred between editors. In PSLC terminology, production rules are equivalent to knowledge components (“Knowledge component,” 2008). 
Support for the knowledge component as the unit of transfer can be found in (Corbett, McLaughlin, & Scarpinatto, 2000). According to the theory of cognitive skill acquisition, the error rate is a function of practice, and it should monotonically decrease with successive opportunities to apply the skill (i.e., the power law of learning). However, when Corbert et al. analyzed the learning curves of 34 students learning how to apply the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, c, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000). 
From the available evidence, we believe that the unit of transfer is the knowledge component. However, as Corbett et al. (2000) demonstrates, some knowledge components are overly general and need to be evaluated empirically. 

==== How can far transfer be supported? ====
Although far transfer is admittedly rare (Detterman, 1993), Gick and Holyoak (1980) and Brown and Kane (1988) demonstrated that it is possible. If it is indeed possible, how can far transfer be supported? One method for supporting far transfer is to look at the cognitive processes and mechanisms that have been identified that support robust learning. Among these are abstract schema induction through analogical comparison (Ross, Holyoak), gap-filling and repair of mental models through the generation of self-explanation inferences (Chi, 2000), meta-cognitive training (Bielaczyc, Pirolli, & Brown, 1995), and self-regulated learning (Pintrich & De Groot, 1990). We chose to focus on analogical comparison because the domain that we have chosen (i.e., translational and rotational kinematics) lends itself to analogical comparison. To illustrate why, consider the equations represented in Table 1. 

Table 1. Equation isomorphisms across two units of physics.
 
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| Eqn. || Translational || Rotational || Assumption
|-
| 1. || <math>\bar{v} = \frac{\Delta s} \Delta t</math>
|-
|
|}
 

Each equation listed in a row is exactly analogous to the equation in its neighboring column. The only difference between the two is that the symbols represent different concepts. For example, in translational kinematics, the vector symbol, , represents the average velocity. Likewise, the vector symbol stands for the average rotational velocity. A similar mapping exists for the other symbols as well: average acceleration ( ) is analogous to average angular acceleration ( ); displacement ( ) is analogous to the angular displacement ( ). 
An additional feature that makes these two units attractive to an analogical-comparison approach is that there are additional concepts to learn besides those listed in Table 1. The additional concepts include radial and tangential acceleration, which do not have analogs in translational motion. This presents an opportunity to measure the existence of accelerated future learning. 
In addition to the content lending itself to analogical comparison, prior research on analogical comparisons suggests that it is an effective instructional intervention because it draws upon the student’s background knowledge. Prior research has shown that students can be guided to construct abstract schemas from making the explicit mapping between two different domains. Educational applications of analogical comparison is in large part inspired by Gentner’s (1983) structure-mapping framework, which states that analogical reasoning is a process whereby an individual creates a mapping between the target (i.e., the unknown domain) and the base (i.e., the known domain). The literal features of the target and base domains are abstracted away to leave only the second-order relations between the objects.
Gentner’s (1983) structure-mapping framework has been used to inform the design of educational interventions. For instance, Ross and Kilbane (1997) attempted to instruct students on solving combination and permutation problems. Specifically, they were interested in measuring the impact of changes made to the variables’ mappings between the study and test problems. For instance, they manipulated whether students solved problems that had identical or dissimilar cover stories. For example, if the study problem was about knights choosing horses for a jousting tournament, a test problem with a similar cover story also used knights and horses. A test problem with a dissimilar cover story, however, used puppies and owners. Both types of test problems, however, reversed the object correspondences such that the horses were now responsible for choosing their riders and puppies choosing their owners. The results from Experiment 2 suggest that the students were able to use the embedded instructional explanations to allow them to see past the superficial features, and make the selection of their variables according to the domain principles. 
Although Experiment 2 of Ross and Kilbane (1997) was effective, there are two major constraints placed on the usefulness of analogical comparison as an effective pedagogical intervention. The first constraint is the observation that students tend to rely too heavily on the surface features of the analogy (see Exper. 1 from Ross & Kilbane, 1997).
The second constraint is that the base domain needs to be well understood by the learner before the mapping to the target domain can make sense (Gentner, Loewenstein, & Thompson, 2003; Kurtz, Miao, & Gentner, 2001). One proposed solution to this limitation is to bootstrap understanding via analogical encoding, which is the idea that students can use an imperfect understanding of two related base domains to understand their deeper structure and principles. To evaluate the efficacy of analogical encoding, Kurtz, Miao, and Genter (2001) asked students to make an explicit correspondence between two images depicting heat transfer. They demonstrate that students, who were asked to make an explicit list of correspondence between the objects of the two scenarios, rated the two disparate situations as more similar than students who were not asked to make systematic comparisons. Unfortunately, Kurtz et al. (2001) did not administer a pretest to diagnose the participant’s initial understanding of the target domain; therefore, it is difficult to assess if the outcome of the analogical encoding was a robust understanding of heat transfer.

Harnessing what you know

2008-10-20T13:49:29Z

Bobhaus: /* Harnessing what you know: The role of analogy in robust learning */

== Harnessing what you know: The role of analogy in robust learning ==
''Robert Hausmann and Timothy J. Nokes''

=== Abstract ===
Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that the material learned in the fourth week of the course is retained and transfers to material taught in the eighth week of the course. This is particularly true for highly structured courses such as physics; however, the empirical literature on learning suggests that far transfer is much more difficult than traditional pedagogy assumes (for reviews, see Bransford, Brown, & Cocking, 2000; Bransford & Schwartz, 1999; Gick & Holyoak, 1983). The goal of the present project is twofold. First, we will use educational data-mining models to identify knowledge components from translational kinematics that fail to transfer to rotational kinematics. Second, we will design an intervention, based upon cognitive principles from self-explanation and analogical comparison, to support knowledge components that fail to transfer.

=== Background and Significance ===
Traditional pedagogy assumes knowledge transfers between problems, units, and even courses; however, the learning literature suggests transfer is rarely observed (Detterman, 1993). Is there transfer between units in a complex science course, such as physics? If so, to what extent?

==== Research Objectives ====
Phase 1. Revise the initial knowledge-component model from the Andes physics tutor for both the translational and rotational kinematics units. 
Phase 2. Develop educational data-mining models to detect the success and failure of the transfer of knowledge components. Student profiles will be defined in an effort to aggregate over individual differences in tutored help-seeking and problem-solving strategies, while still being sensitive to them. 
Phase 3. Design an instructional intervention, based on cognitive science principles, to facilitate transfer between units. The format of the intervention will be designed around the literature on analogical comparison and self-explanation. The content of the intervention will be based on the revised knowledge-component model, the identification of failed knowledge-component transfer, and student profiles. 

==== Hypotheses ====

H1: The learning curves from translational kinematics knowledge components can predict the error rates for rotational kinematics. 
H2: Educational interventions that draw upon prior knowledge, such as analogical comparison and self-explanation, can support knowledge components that fail to transfer between translational and rotational kinematics. 

=== Prior Work ===
==== Near and Far Transfer ====
Knowledge transfer is a core assumption built into the pedagogy of most educational programs from K-12 to college. It is assumed that learning is cumulative and that advanced courses will build on the knowledge and skills acquired in the introductory and foundational courses. This is especially true for the STEM disciplines, where students take a highly structured sequence of courses. To illustrate this assumption, consider the case of introductory physics, which is typically split among two semesters. In a traditional curriculum, the first semester covers Mechanics, and the second semester covers Electricity and Magnetism. Instructors of Physics II assume that the material learned in the first semester is retained and can be applied to the problems related to the motion of a charged particle in an electric or magnetic field. Moreover, most pedagogy assumes within-class transfer as well. That is, topics and concepts taught later in a course build upon and extending those taught earlier. 
Unfortunately, research on human cognition has shown that knowledge transfer (especially far transfer to novel contexts and applications) is much more rare than traditional pedagogy assumes (for reviews, see Barnett & Ceci, 2002; Bransford, Brown, & Cocking, 2000). For example, in a classic study on transfer, Gick and Holyoak (1980) asked participants to solve a difficult insight problem (i.e., the solution rate was 8%). Before solving this difficult problem, all of the participants read a story that proposed an analogous solution. Half of the participants received a hint that the story will help with the solution, whereas the other half of the students did not receive a hint. The results were clear. The solution rate was much lower (i.e., 20%) for the participants who did not receive any hints, whereas those who received hints demonstrated a dramatic increase in their solution rate (i.e., 92%). These results suggest that spontaneous far transfer is difficult for students to implement. 
However, because a fifth of the students were able to spontaneously transfer their knowledge of one domain to another, Gick and Holyoak (1980) demonstrate that spontaneous far transfer is indeed possible. With the appropriate scaffolding in place, it becomes quite likely. This is also true for children learning authentic science material. For instance, Brown and Kane (1988) taught pre-school children animal defense mechanisms such as mimicry. The children’s ability to transfer the concept of mimicry to other animals depended crucially on their depth of understanding. That is, if the child understood mimicry at the level of the causal structure, then they were more likely to demonstrate transfer; whereas if the child was only imitating the behavior of the teacher, then they failed to transfer the concept. 

==== What transfers? ====
Often, the debate surrounding whether far transfer is tenable must address the issue of the unit of analysis. In other words, what transfers? Several hypotheses have been posited, including the doctrine of formal discipline from antiquity, Thorndike’s theory of identical elements, and Singley and Anderson’s (1989) identical-productions theory of transfer. The formal discipline theory implicated entire domains of knowledge were the units of analysis. For instance, politicians would be well advised to learn mathematics because it will cause them to be quicker thinkers (Lehman, Lempert, & Nisbett, 1988). In other words, the mind is analogous to a muscle that, when exercised properly, will increase in strength.
However, early psychologists took issue with the doctrine of formal discipline and challenged it on empirical grounds. Thorndike and Woodword (1901a; 1901b; 1901c) demonstrated, in an impressive series of studies, that transfer could only be expected if the two tasks shared “common elements.” For example, receiving training on estimating the area of a rectangle did not reduce the error rate of estimating the area of a different shape (e.g. triangle). 
Similar findings have been demonstrated with abstract reasoning tasks. For instance, Wason (1968) developed a deceptively simple task to assess an individual’s ability to reason about a bi-conditional rule. First-year psychology and statistics students were asked to evaluate the following rule: “If there is a D on one side of any card, then there is a 3 on its other side.” Then they were shown four cards that had a symbol on one side and another symbol in brackets indicating the contents of the back of the card. The cards were: D(3), 3(K), B(5), 7(D). The cards were placed in random order in front of the participant, and the experimenter pointed to each card and asked if that card could be used to determine if the rule was true or false. Collapsing across conditions, only 14.7% of the participants were able to correctly identify the cards that tested the veracity of the rule.
In a follow-up study, Cheng, Holyoak, Nisbett, and Oliver (1986) investigated the conditions under which formal training can enhance performance on abstract tasks, such as the Wason 4-card selection task. They found, after an entire semester of instruction on logic, there was no difference in the error rate on the Wason task (Exper. 2; p. 306). Even more to the point, Cheng et al. created their own training materials that were specifically designed to improve logical reasoning. Again, they found that performance on the Wason four-card task was not improved by their customized formal instruction alone (Exper. 1). 
From the available evidence, it appears that entire disciplines are not the unit of transfer, nor is the proposal of common elements of transfer specific enough to make predictions about what exactly transfers between two learning situations. A more specific theory of what constitutes an “element” is Singley and Anderson’s (1989) hypothesis that production rules, or skills, are the unit of transfer. In their analysis of learning how to use text editors, they demonstrated that the surface features can vary substantially, yet the production rules that compose the cognitive skill are transferred between editors. In PSLC terminology, production rules are equivalent to knowledge components (“Knowledge component,” 2008). 
Support for the knowledge component as the unit of transfer can be found in (Corbett, McLaughlin, & Scarpinatto, 2000). According to the theory of cognitive skill acquisition, the error rate is a function of practice, and it should monotonically decrease with successive opportunities to apply the skill (i.e., the power law of learning). However, when Corbert et al. analyzed the learning curves of 34 students learning how to apply the quadratic formula, they found an inexplicable jump in the error rate at the fourth opportunity to apply the quadratic knowledge component (see Fig. 8 from Corbett et al., 2000). To address this anomaly, they conducted a fine-grained analysis of the problems and discovered that on some of the problems, the constant term, c, was zero, and on other problems, the constant term was a positive integer. They inferred that the knowledge component APPLY-THE-QUADRATIC-FORMULA was an overly general rule and should be decomposed into two smaller skills. After making the decomposition, the error rate aligned with the theoretical prediction (see Fig. 9 from Corbett et al., 2000). 
From the available evidence, we believe that the unit of transfer is the knowledge component. However, as Corbett et al. (2000) demonstrates, some knowledge components are overly general and need to be evaluated empirically. 

==== How can far transfer be supported? ====
Although far transfer is admittedly rare (Detterman, 1993), Gick and Holyoak (1980) and Brown and Kane (1988) demonstrated that it is possible. If it is indeed possible, how can far transfer be supported? One method for supporting far transfer is to look at the cognitive processes and mechanisms that have been identified that support robust learning. Among these are abstract schema induction through analogical comparison (Ross, Holyoak), gap-filling and repair of mental models through the generation of self-explanation inferences (Chi, 2000), meta-cognitive training (Bielaczyc, Pirolli, & Brown, 1995), and self-regulated learning (Pintrich & De Groot, 1990). We chose to focus on analogical comparison because the domain that we have chosen (i.e., translational and rotational kinematics) lends itself to analogical comparison. To illustrate why, consider the equations represented in Table 1. 

Table 1. Equation isomorphisms across two units of physics.

Eqn. Translational Rotational Assumption
1

2

3

when ti = 0
4

when si or i = 0
5

Each equation listed in a row is exactly analogous to the equation in its neighboring column. The only difference between the two is that the symbols represent different concepts. For example, in translational kinematics, the vector symbol, , represents the average velocity. Likewise, the vector symbol stands for the average rotational velocity. A similar mapping exists for the other symbols as well: average acceleration ( ) is analogous to average angular acceleration ( ); displacement ( ) is analogous to the angular displacement ( ). 
An additional feature that makes these two units attractive to an analogical-comparison approach is that there are additional concepts to learn besides those listed in Table 1. The additional concepts include radial and tangential acceleration, which do not have analogs in translational motion. This presents an opportunity to measure the existence of accelerated future learning. 
In addition to the content lending itself to analogical comparison, prior research on analogical comparisons suggests that it is an effective instructional intervention because it draws upon the student’s background knowledge. Prior research has shown that students can be guided to construct abstract schemas from making the explicit mapping between two different domains. Educational applications of analogical comparison is in large part inspired by Gentner’s (1983) structure-mapping framework, which states that analogical reasoning is a process whereby an individual creates a mapping between the target (i.e., the unknown domain) and the base (i.e., the known domain). The literal features of the target and base domains are abstracted away to leave only the second-order relations between the objects.
Gentner’s (1983) structure-mapping framework has been used to inform the design of educational interventions. For instance, Ross and Kilbane (1997) attempted to instruct students on solving combination and permutation problems. Specifically, they were interested in measuring the impact of changes made to the variables’ mappings between the study and test problems. For instance, they manipulated whether students solved problems that had identical or dissimilar cover stories. For example, if the study problem was about knights choosing horses for a jousting tournament, a test problem with a similar cover story also used knights and horses. A test problem with a dissimilar cover story, however, used puppies and owners. Both types of test problems, however, reversed the object correspondences such that the horses were now responsible for choosing their riders and puppies choosing their owners. The results from Experiment 2 suggest that the students were able to use the embedded instructional explanations to allow them to see past the superficial features, and make the selection of their variables according to the domain principles. 
Although Experiment 2 of Ross and Kilbane (1997) was effective, there are two major constraints placed on the usefulness of analogical comparison as an effective pedagogical intervention. The first constraint is the observation that students tend to rely too heavily on the surface features of the analogy (see Exper. 1 from Ross & Kilbane, 1997).
The second constraint is that the base domain needs to be well understood by the learner before the mapping to the target domain can make sense (Gentner, Loewenstein, & Thompson, 2003; Kurtz, Miao, & Gentner, 2001). One proposed solution to this limitation is to bootstrap understanding via analogical encoding, which is the idea that students can use an imperfect understanding of two related base domains to understand their deeper structure and principles. To evaluate the efficacy of analogical encoding, Kurtz, Miao, and Genter (2001) asked students to make an explicit correspondence between two images depicting heat transfer. They demonstrate that students, who were asked to make an explicit list of correspondence between the objects of the two scenarios, rated the two disparate situations as more similar than students who were not asked to make systematic comparisons. Unfortunately, Kurtz et al. (2001) did not administer a pretest to diagnose the participant’s initial understanding of the target domain; therefore, it is difficult to assess if the outcome of the analogical encoding was a robust understanding of heat transfer.

Harnessing what you know

2008-10-20T13:43:37Z

Bobhaus: New page: == Harnessing what you know: The role of analogy in robust learning == ''Robert Hausmann and Timothy J. Nokes'' === Abstract === Knowledge transfer is a core assumption built into the pe...

Plateau study

2008-10-10T18:08:43Z

Bobhaus: /* Independent variables */

== The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction ==
''Robert G.M. Hausmann, Brett van de Sande, & Kurt VanLehn''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Robert G.M. Hausmann (Pitt), Brett van de Sande (Pitt), & Kurt VanLehn (Pitt)
|-
| '''Other Contributers''' || Tim Nokes (Pitt)
|-
| '''Study Start Date''' || Feb. 21, 2007
|-
| '''Study End Date''' || March 07, 2008
|-
| '''LearnLab Site''' || University of Pittsburgh (Pitt)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = 21
|-
| '''Total Participant Hours''' || 42 hrs.
|-
| '''DataShop''' || Anticipated: June 1, 2008
|}
 

=== Abstract ===
This study will test the hypothesis that, as the degree of interaction increases, the learning gains first increase then level off. The empirical pattern is called the interaction plateau. We will test the interaction plateau by comparing human tutoring (high interaction) to the Andes intelligent tutoring system (medium interaction) to solving problems with feedback on correct answers (low interaction). We anticipate finding High = Medium > Low for learning gains from these 3 types of physics instruction.

=== Background and Significance ===
The goal of this research is to test the interaction plateau hypothesis by comparing human physics tutors to two other forms of physics instruction. The first type of instruction is coached problem solving, via an intelligent tutoring system for physics called Andes. Andes has been shown to produce large learning gains when compared to normal classroom instruction (Kurt VanLehn et al., 2005). The second type of instruction is computer-aided instruction (CAI), which has only shown to be moderately effective. In the present study, the feedback and guidance provided by the full-blown version of Andes will be removed so that the students are responsible for all of the problem-solving steps.

A literature review summarizes the background knowledge: VanLehn, K. (in prep.) The interaction plateau: Is highly interactive, one-on-one, natural language tutoring as effective as simpler forms of instruction?

Reif and Scott (1999) found an interaction plateau when they compared human tutoring, a computer tutor and low-interaction problem solving. All students in their experiment were in the same physics class; the experiment varied only the way that the students did their homework. One group of 15 students did their physics homework problems individually in a six-person room where “two tutors were kept quite busy providing individual help” (ibid, pg. 826). Another 15 students did their homework on a computer tutor that had them either solve a problem or study a solution. When solving a problem, students got immediate feedback and hints on each step. When studying a problem, they were shown steps and asked to determine which one(s) were incorrect. This forced them to derive the steps. Thus, this computer tutor counts as step-based instruction. The remaining 15 students merely did their homework as usual, relying on the textbook, their friends and the course TAs for help. The human tutors and the computer tutors produced learning gains that were not reliably different, and yet both were reliably larger than the low-interaction instruction provided by normal homework (d = 1.31 for human tutoring; d = 1.01 for step-based computer tutoring).

In a series of experiments, VanLehn et al. (2007) taught students to reason out answers to conceptual physics questions such as: “As the earth orbits the Sun, the sun exerts a gravitational force on it. Does the earth also exert a force on the sun? Why or why not?” In all conditions of the experiment, students first studied a short textbook, then solved several training problems. For each problem, the students wrote an short essay-long answer, then were tutored on its flaws, then read a correct, well-written essay. Students were expected to apply a certain set of concept in their essays—these comprised the correct steps. The treatments differed in how they tutored students when the essay lacked a step or had an incorrect step. There were four experimental treatments: (1) Human tutors who communicated via a text-based interface with student; (2) Why2-Atlas and (3) Why2-AutoTutor, both of which were natural language computer tutors designed to approach human tutoring; and (4) a simple step-based computer tutor that “tutored” a missing or incorrect step by merely display text that explained what the correct step was. A control condition had students merely read passages from a textbook without answering conceptual questions. The first 3 treatments all count as natural tutoring, so according to the interaction plateau, they should all have the same learning gains as the simple step-based tutoring system. All four experimental conditions should score higher than the control condition, as it is classified as read-only studying of text. The four experimental conditions are not reliably different, and they all were higher than the read-only studying condition by approximately d = 1.0. Thus, the results of experiments 1 and 2 support the interaction plateau.

In a series of experiments, (Evens & Michael, 2006) tutored medical students in cardiovascular physiology. All students were first taught the basics of the baroreceptor reflex which controls human blood pressure. They were then given a training problem wherein an artificial pacemaker malfunctions and students must fill out a spreadsheet whose rows denoted physiological variables (e.g., heart rate; the blood volume per stroke of the heart, etc.) and whose column denoted time periods. Each cell was filled with a +, - or 0 to indicate that the variable was increasing, decreasing or constant. Each such entry was a step. The authors first developed a step-based tutoring system, CIRCSIM, that presented a short text passage for each incorrectly entered step. They then developed a sophisticated natural language tutoring system, CIRCSIM-tutor, which replaced the text passages with human-like typed dialogue intended to remedy not just the step but the concepts behind the step as well. They also used a read-only studying condition with an experimenter-written text, and they included conditions with expert human tutors interacting in typed text with students. The treatments that count as Natural Tutoring (the expert human tutors and CIRCSIM-tutor) tied with each other and with the step-based computer tutor (CIRCSIM). The only conditions were learning gains were significantly different were the read-only text studying treatments. This pattern is consistent with the interaction plateau.

There are two important reasons for conducting this study. The first is theoretical and the second is applied.

1. Theoretical: No study, to our knowledge, has directly compared human tutoring, an intelligent tutoring system, and computer-aided instruction in a single study.

2. Applied: Testing the interaction plateau hypothesis will help us better understand both how to construction educational software and how to train human tutors.

=== Glossary ===
* [[Andes]]

See [[:Category:Plateau Study|Plateau Study Glossary]]

=== Research question ===
How is [[robust learning]] affected by the amount of interaction the student has with the learning environment?

=== Independent variables ===
This study used one independent variable, Interactivity, with three levels. The first level was the most restricted amount of interaction. The students interacted with the learning environment by typing their answers into an text-box, which was embedded in an Answer-only version of Andes. The answer box gave correct/incorrect feedback on the answer, and gave a limited range of hints (e.g., "You forgot the units on your answer."). The second level of interactivity was the full-blown version of Andes. Students received correct/incorrect feedback on each step, and they were free to request hints on each step. Finally, the highest level of interactivity was one-on-one, human tutoring. Tutors were recruited from the physics department, and they were either graduate students or faculty in the Physics program at the University of Pittsburgh.

Figure 1. An example from the Human Tutoring condition 
{| border="1"
| [[Image:Human_tutoring.JPG]] 
|}

Figure 2. An example from the full Andes condition 
{| border="1"
| [[Image:Full_Andes.JPG]] 
|}

Figure 3. An example from the Answer-only condition 
{| border="1"
| [[Image:Answer_only.JPG]] 
|}

=== Hypothesis ===
This experiment implements the following instructional principle: [[Interaction plateau]]

=== Dependent variables ===
* [[Normal post-test]]
** ''Qualitative Assessment'':
** ''Quantiative Assessment'':

=== Results ===
'''Procedure'''
The participants were asked to complete a series of tasks. Upon granting informed consent, participants were then administered a pretest, which consisted of 13 multiple-choice conceptual physics questions. In addition, they were also given two quantitative problems. Both problems were decomposed into three sub-problems. They were given a total of 20 minutes to complete the pretest.

After completing the pretest, participants were then given a short text to study. They were not given any instructions as to how to study the text. That is, the undergraduate student volunteers studied the short text using the same learning strategies that they use when studying their text for class, either to solve homework problems or take an exam. They were given 15 minutes to study the text.

Next, the experimental intervention took place. Students assigned to the Answer-only condition were given an "answer booklet" in which they were encouraged to write their solution to the problem. The problems were presented both as a problem statement in the Andes tutoring system, as well as on paper. Students assigned to the Andes condition were given the exact same problems. The main difference between these conditions was the help and feedback that they received. The Answer-only condition only received feedback on their final numerical answer. If omitted, the system would prompt the student to include the correct units. They also received the red/green flag feedback on the answer. In contrast, the Andes condition received both flag feedback and hints were made available on each problem solving step.

Students assigned to the human tutoring condition were given the same problems to solve. The tutor and student worked face-to-face, using a dry-erase board to display their work. The interaction was audio- and video-taped for later analysis. The experimental intervention lasted 45 minutes.

After tutored problem solving, all of the participants then completed a post-test. The post-test consisted of the same 13 multiple-choice problems. However, the quantitative problems were not identical to the pretest. Instead, they drew upon the same concepts and principles, but the surface form of the problems was different. Participants were given 20 minutes to solve the post-test problems.

* [[Normal post-test]]: ''Qualitative Assessment''

At this point, we have only analyzed the results from the multiple choice test. To measure the change in students' qualitative understanding, we used the following measure: gain = (post - pre)/(100% - pre).

 <center>[[Image:mc_normgain.JPG]] '''Figure 1.''' The normalized gain score for the multiple-choice problems. </center>

Instead of a plateau, the results look more like an "interaction valley." We were curious why the pattern of results showed the Andes condition performing marginally worse than the Human tutoring condition (Fisher's LSD, ''p'' = .09). We hypothesized that the interface of the Andes interface takes more time to learn than the Answer-only version of Andes. To test this hypothesis, we looked at the number of students, in each condition, that attempted to solve each of the four problems during the learning phase of the experiment. The learning phase was when the experimental manipulation took place. Thus, we would expect that the Andes condition would have attempted fewer problems if they are taking longer to learn the interface. The figure below summarizes the percentage of students who attempted each problem.

 <center>[[Image:mc_attempts.JPG]] '''Figure 2.''' The number of problems attempted by each condition during the learning phase of the experiment. </center>

It is extremely clear that the Andes condition was unable to solve the same number of problems as the other two conditions.

=== Explanation ===
=== Further Information ===
==== Annotated bibliography ====
*

==== References ====
# VanLehn, K. (in prep.) The interaction plateau: Is highly interactive, one-on-one, natural language tutoring as effective as simpler forms of instruction?
# Reif, F., & Scott, L. A. (1999). Teaching scientific thinking skills: Students and computers coaching each other. ''American Journal of Physics, 67''(9), 819-831.
# VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are tutorial dialogues more effective than reading? ''Cognitive Science, 31''(1), 3-62.
# Evens, M., & Michael, J. (2006). ''One-on-one Tutoring By Humans and Machines.'' Mahwah, NJ: Erlbaum.

==== Connections ====
*

==== Future plans ====

File:Human tutoring.JPG

2008-10-10T18:08:37Z

Bobhaus: uploaded a new version of "Image:Human tutoring.JPG"

File:Full Andes.JPG

2008-10-10T18:07:39Z

Bobhaus:

File:Human tutoring.JPG

2008-10-10T18:07:00Z

Bobhaus:

File:Answer only.JPG

2008-10-10T18:06:46Z

Bobhaus:

Self-explanation: Meta-cognitive vs. justification prompts

2008-10-10T17:36:38Z

Bobhaus: /* Independent variables */

== Self-explanation: Meta-cognitive vs. justification prompts ==
''Robert G.M. Hausmann, Brett van de Sande, Sophia Gershman, & Kurt VanLehn''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Robert G.M. Hausmann (Pitt), Brett van de Sande (Pitt), Sophia Gershman (WHRHS), & Kurt VanLehn (Pitt)
|-
| '''Other Contributers''' || Tim Nokes (Pitt)
|-
| '''Study Start Date''' || Sept. 1, 2007
|-
| '''Study End Date''' || Aug. 31, 2008
|-
| '''LearnLab Site''' || Watchung Hills Regional High School (WHRHS)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = 75
|-
| '''Total Participant Hours''' || 150 hrs.
|-
| '''DataShop''' || Anticipated: June 1, 2008
|}
 

=== Abstract ===
The literature on studying examples and text in general shows that students learn more when they are prompted to self-explain the text as they read it. Experimenters have generally used two types of prompts: meta-cognitive and justification. An example of a meta-cognitive prompt would be, "What did this sentence tell you that you didn't already know?" and an example of a justification prompt would be, "What reasoning or principles justifies this sentence's claim?" To date, no study has included both types of prompts, and yet there are good theoretical reasons to expect them to have differential impacts on student learning. This study will directly compare them in a single experiment using high schools physics students.

=== Background and Significance ===
The self-explanation effect has been empirically demonstrated to be an effective learning strategy, both in the laboratory and in the classroom. The effect sizes range from ''d'' = .74 – 1.12 in the lab for difficult problems (Chi, DeLeeuw, Chiu, & LaVancher, 1994; McNamara, 2004) to ''d'' = .44 – .92 in the classroom (Hausmann & VanLehn, 2007). However, both the amount and quality of spontaneously produced self-explanations is highly variable (Renkl, 1997). To increase both the likelihood and quality, different prompting procedures have been designed to solicit student-generated explanations. However, an open question is how to structure the learning environment to maximally support learning from self-explanation. One method to support robust learning is to design instructional prompts that increase the probability that students will frequently generate high-quality self-explanations. What counts as a “high-quality self-explanation?”

The answer to that question may depend on the type of knowledge to be learned. Knowledge can be categorized into two types, either procedural or declarative knowledge. In physics, students often learn the procedural skill of solving problems by studying examples. An example is a solution to a problem, which is derived in a series of steps. An example step contains either an application of a physics principle or mathematical operator. The transition from one step to the next can be justified by a reason consisting of the applicable principle or operator. Therefore, an effective prompt for procedural learning asks the student to justify each step of an example with a domain principle or operator. For the purposes of this proposal, we shall call this type of prompt “justification prompts.”

Contrast a high-quality self-explanation from problem solving in physics with an explanation from a declarative domain (i.e., the human circulatory system). In this domain, the student’s task is to develop a robust mental model of a physical system. Instead of a solution example broken down by steps, the information is presented as a text, with each sentence presented separately. When each sentence is read, the student’s goal is to revise or augment his or her initial mental model (Chi, 2000). A high-quality explanation in this domain requires the student to consider the relationship between the structure, behavior, and function of the various anatomical features of the circulatory system. When a student begins reading about the heart, she rarely (if ever) comes to the task as a blank slate. More likely, the student has an initial mental model that is flawed in some way. Therefore, the student must revise her initial mental model to align itself with the content of the text. This is generally not an easy task because the reader is required to use her prior knowledge to comprehend the text, while simultaneously revise that same knowledge. In this case, a high-quality self-explanation may consist of reflecting on one’s own understanding, comparing it to the target material, explaining the discrepancies between the two, and revising the mental model. Therefore, we shall refer to prompts that encourage this type of behavior as “meta-cognitive prompts.”

Thus, different types of self-explanation prompts may lead to different learning outcomes. Justification-based prompts may inspire more gap-filling activities, while meta-cognitive prompts may evoke more mental model repair. While both types of prompting techniques have been used in prior research on self-explaining, what remains to be explored, however, is a systematic exploration of the differential impact prompting for justifications or meta-cognitive activities on robust learning.

=== Glossary ===
* [[Jointly constructed explanation]]
* [[Prompting]]

See [[:Category:Hausmann_Study2|Hausmann_Study2 Glossary]]

=== Research question ===
How is [[robust learning]] affected by justification-based vs. meta-cognitive prompts for self-explanation?

=== Independent variables ===
Only one independent variable, with three levels, was used:
* Prompt Type: meta-cognitive prompts vs. justification-based prompts vs. attention-focusing prompts

[[Prompting]] for an explanation was intended to increase the probability that the individual or dyad will traverse a useful learning-event path. Meta-cognitive prompts were designed to increase the students' awareness of their developing knowledge. The justification-based prompts were designed to motivate students to explicitly articulate the principle needed to solve a particular problem. Finally, the attention-focusing prompts were designed as a set of control prompts. The purpose was to steer their attention to the examples that they are studying, without motivating any particular type of active cognitive processing.

Figure 1. An example from the Justification-based condition 
[[Image:JUST.JPG]] 

Figure 2. An example from the Meta-cognitive condition 
[[Image:META.JPG]] 

Figure 3. An example from the Attention-focusing condition 
[[Image:ATTN.JPG]] 

=== Hypothesis ===
This experiment implements the following instructional principle: [[Prompted Self-explanation]]

=== Dependent variables ===
* [[Normal post-test]]
** ''Near transfer, immediate'': During training, [[worked examples]] alternated with problems, and the problems were solved using [[Andes]]. Each problem was similar to the [[worked examples|example]] that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). The log data were analyzed and assistance scores (sum of errors and help requests, normalized by the number of transactions) were calculated.

* [[Robust learning]]
** ''[[Long-term retention]]'': On the student’s regular mid-term exam, one problem was similar to the training. Since this exam occurred a week after the training, and the training took place in just under 2 hours, the student’s performance on this problem is considered a test of [[long-term retention]].
** ''Near and far [[transfer]]'': After training, students did their regular homework problems using [[Andes]]. Students did them whenever they wanted, but most completed them just before the exam. The homework problems were divided based on similarity to the training problems, and assistance scores were calculated.
** ''[[Accelerated future learning]]'': The training was on electrical fields, and it was followed in the course by a unit on magnetic fields. Log data from the magnetic field homework was analyzed as a measure of acceleration of future learning.

=== Results ===
'''Procedure'''
Participants were randomly assigned to condition. The first activity was to train the participants in their respective explanation activities. They read the instructions to the experiment, presented on a webpage, followed by the prompts used after each step of the example.

All of the participants were enrolled in a year-long, high-school physics course. The task domain, electrodynamics, was taught at the beginning of the Spring semester. Therefore, all of the students were familiar with the Andes physics tutor. They did not need any training in the interface. Unlike our previous [[Hausmann_Study2 | lab experiment]], they did not solve a warm-up problem. Instead, they started the experiment with a fairly complex problem.

Once they finished, participants then watched a video solving an ''isomorphic problem''. Note that this procedure is slightly different from previous research, which used examples presented before solving problems (e.g., Sweller & Cooper, 1985; Exper. 2). The videos decomposed into steps, and students were prompted to explain each step. The cycle of explaining examples and solving problems repeated until either 3 problems were solved or the class period was over. The problems were designed to become progressively more complex.

The results reported below are from a median split conducted on the assistance score on the first problem. The students who were below the median are included in the analyses because the strong prior-knowledge students were unaffected by the different types of prompts.

* [[Normal post-test]]
** ''Near [[transfer]], immediate'': The prompting for justifications demonstrating lower normalized assistance scores for later problems than the other two conditions. <center>[[Image:assist_results.JPG]]</center>
** In addition, the justification-based prompting condition demonstrated faster solution times than the other two conditions for the last problem. <center>[[Image:time_results.JPG]]</center>

* [[Robust learning]]
** ''Near transfer, retention'': to be collected and analyzed.
** ''Near and far transfer'': to be collected and analyzed.
** ''[[Accelerated future learning]]'': to be collected and analyzed.

=== Explanation ===
Data analyses are preliminary and ongoing. Thus, the results reported above are speculative at this point. However, the trends suggest that justification prompting for physics may provide a learning advantage for later, more complex and difficult problems. One reason for the superiority of justification-based prompting is related to the match between the instructional materials and the cognitive task demands. That is, there are two sequential learning events that occur during the experiment. The first is learning from problem solving. In this case, the student must search the problem space in order to arrive at a fully specified solution. In so doing, he or she will most likely access and apply principles from physics. There are multiple competing features in the environment that cue the retrieval of these principles. The valid features that signal the correct application of a principle are called the "applicability conditions." During example studying, the student is released from the task of accessing the principles and is instead asked to generation the conditions of applicability. Generating the justification for a problem-solving action is commensurate with this task.

Prompting for meta-cognitive reflection and evaluation, however, may not fit the task as well. Instead, students who do not have strong prior knowledge may find the lack of structure unhelpful. Weaker students may need a more structured learning environment.

The attention prompts are ambiguous because 2 of them may inspire the student to self-explain. A close analysis of the audio portions of the data that were collected during the example-studying phase may allow for a direct test of this hypothesis.

=== Further Information ===
==== Annotated bibliography ====
*

==== References ====
# Chi, M. T. H. (2000). Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In R. Glaser (Ed.), Advances in instructional psychology (pp. 161-238). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. [http://www.pitt.edu/~chi/papers/advances.pdf]
# Chi, M. T. H., DeLeeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, 439-477. [http://www.pitt.edu/~chi/papers/ChideLeeuwChiuLaVancher.pdf]
# Hausmann, R. G. M., & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial intelligence in education: Building technology rich learning contexts that work (Vol. 158, pp. 417-424). Amsterdam: IOS Press. [http://learnlab.org/uploads/mypslc/publications/hausmannvanlehn2007_final.pdf]
# McNamara, D. S. (2004). SERT: Self-explanation reading training. Discourse Processes, 38(1), 1-30. [http://www.leaonline.com/doi/abs/10.1207/s15326950dp3801_1?journalCode=dp]
# Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. Cognitive Science, 21(1), 1-29. [http://www.leaonline.com/doi/pdf/10.1207/s15516709cog2101_1]

==== Connections ====
*

==== Future plans ====

File:JUST.JPG

2008-10-10T17:36:11Z

Bobhaus:

File:META.JPG

2008-10-10T17:30:13Z

Bobhaus:

File:ATTN.JPG

2008-10-10T17:30:01Z

Bobhaus:

Hausmann Diss

2008-10-10T17:02:34Z

Bobhaus: /* Independent variables */

== Elaborative and critical dialog: Two potentially effective problem-solving and learning interactions ==
''Robert G.M. Hausmann and Michelene T.H. Chi''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Robert G.M. Hausmann & Michelene T.H. Chi
|-
| '''Study Start Date''' || Sept. 20, 2004
|-
| '''Study End Date''' || April 28, 2005
|-
| '''LearnLab Site''' || none
|-
| '''LearnLab Course''' || none
|-
| '''Number of Students''' || ''N'' = 136
|-
| '''Total Participant Hours''' || 68 hrs.
|-
| '''DataShop''' || no
|}
 

=== Abstract ===

Recent research on peer dialog suggests that some dialog patterns are more strongly correlated with learning than others. A peer dialog, which is a subordinate category of interactive communication, occurs when two novices work together to collaboratively learn a set of [[knowledge component]]s, solve a problem, or both. Two aspects of peer dialog that have been shown to be correlated with learning are elaboration and constructive criticism. Elaboration can be defined as a conditionally relevant contribution that significantly develops another person’s idea. Constructive criticism is defined as either a request for justification or an evaluation of an idea. The primary goal for this project was to move beyond correlating dialog patterns with outcomes by training the participants to interact in specific ways.

Participants were randomly assigned to one of four conditions: elaborative dyads, critical dyads, control dyads, and individuals. They were asked to solve a design problem, which was to optimize the design of a pre-existing bridge structure. Participants iteratively edited their design, analyzed its cost and effectiveness, and discussed their analyses to formulate their next modification. This process continued for thirty minutes, after which a post-test measuring both text-explicit and deep knowledge was administered.

The results indicated that the generated the same number of critical statements as control dyads; therefore, the critical condition was collapsed into the control condition. Alternatively, the elaborative condition generated better designs and learned more deep knowledge than the control condition. The elaborations led to shorter negotiations about what design modification to try next, so more designs were tried. These students thus sampled more of the underlying design space. This may also explain their increased learning because more appropriate [[learning events]] occurred. The problem-solving and learning outcomes also suggest that training individuals to elaborate may have been easier than asking them to produce evaluative statements.

=== Background and Significance ===

Past research on collaborative problem solving and learning has painted a fairly consistent picture of both its costs and benefits. For instance, [[collaboration]] seems to be an effective educational intervention; however, not all collaborative dialogs lead to positive outcomes. Instead, only certain dialog patterns tend to result in strong learning gains. For example, generating explanations tends to be associated with understanding (Chi, Bassok, Lewis, Reimann, & Glaser, 1989) while paraphrasing does not (Hausmann & Chi, 2002). Therefore, one of the goals of the learning sciences is to identify dialog patterns that tend to produce strong learning gains (Dillenbourg, Baker, Blaye, & O'Malley, 1995). One such candidate is elaborative dialogs (Brown & Palincsar, 1989). Elaboration is a likely candidate because it has been shown to support individual learning. Why is it the case that elaboration is an effective problem solving interaction? Critical interactions are another useful collaborative dialog. For example, the argumentation literature has shown that argumentation and counter-argument are related to a deep understanding of a domain (Leitao, 2003). However, an open question is how elaborative and critical interactions compare.

=== Glossary ===
* '''[[Elaborative interaction]]'''
* '''[[Critical interaction]]'''
See [[:Category:Hausmann_Diss|Hausmann_Diss Glossary]]

=== Research question ===

* Can students be trained to collaborate in specific ways? If so, what is the effect of collaborative training on problem-solving performance and learning?
* Do elaborative or critical interactions lead to better problem solving and/or learning than unscripted interactions?
* Why do elaborative dialogs lead to efficient problem solving and deep learning?

=== Independent variables ===

* Collaboration training: elaboration vs. critical vs. control vs. individual

Figure 1. An example from the Elaboration condition 
{| border="1"
| [[Image:IER.JPG]] 
|}

Figure 2. An example from the Critical condition 
{| border="1"
| [[Image:ICR.JPG]] 
|}

Figure 3. An example from the Control Paraphrase condition 
{| border="1"
| [[Image:Diss_Control.JPG]] 
|}

Figure 4. An example from the Individual Paraphrase condition 
{| border="1"
| [[Image:IND.JPG]] 
|}

=== Hypothesis ===

The Interactive Communication cluster assumes that different types of interactions lead to different types of learning.

Elaborative dialog is hypothesized to enhance learning and problem solving by increasing the specification of another person’s message. To illustrate the hypothesis, consider an example from the domain in which this study was conducted: [http://bridgecontest.usma.edu/ bridge design]. Suppose one member of the dyad suggests they decrease the cross-sectional diameter of the members. The second member takes up this suggestion and extends it by proposing to only reduce the diameter of the vertical members. The verb "to change" requires the assignment of three variables: 1. An object to be modified, 2. The old property, and 3. The new property. Therefore, problem solving can be more efficient if the dyad elaborates the ideas by filling open variables. In contrast to elaboration, one member of the dyad could ask for clarification of the first member's idea (i.e., to request that the individual fill his or her own unfilled variable assignments). If the dyad is efficient, then they will be able to expose themselves to a wider array of knowledge components embedded in the simulated environment, thereby enhancing their learning from [[collaboration]].

Critical dialogs are hypothesized to enhance problem solving by avoiding the space of designs that are ineffective, thereby saving time testing fewer bridges. The process of evaluation may also increase learning because the knowledge components used in this task may be more highly specified for particular applications. For instance, the participants are told that shorter bars are stronger under compression. Until they have seen this knowledge component in an actual design, they might not appreciate its importance. Thus, the knowledge component is reified under a concrete application.

This experiment implements the following instructional principle: [[Collaboration scripts]]

=== Dependent variables ===
==== Learning ====

* ''[[Normal post-test]], Near transfer, immediate'': Text-explicit learning was defined as the acquisition of information explicitly stated in the text. Text-explicit learning was measured by administering identical pre- and post-tests. Standardized gain scores were used to measure learning.

* ''Far [[transfer]], immediate'': Deep, inferential learning was defined as concepts that were not explicitly stated, and thus needed to be inferred from reading the text or interacting with the simulation. Inferential learning was also measured by administering identical pre- and post-tests, using standardized gain scores.

==== Problem Solving ====

* ''Iterations'': the number of bridges tested. Measuring the number of designs tested served as a proxy variable for productivity. That is, if the pair is able to make a suggestion and agree quickly, then they will be able to test more designs, signalling higher levels of productivity.

* ''Optimization score'': the summation of the stress-to-strength ratios for each member, which was then divided by the total number of members:

<math>\frac{\sum_{n=1}^i}{\frac{stress_{i}}{strength_{i}}}{n}</math>

where n = the number of members per bridge, stress is the amount of force loaded on the i-th member of the bridge, and strength is the maximum loading the i-th member can withstand before failure. The optimization score represents the average load per member; thus, higher values indicate better optimized designs.

* ''Savings score'': the amount of money saved was calculated by subtracting the price of their final bridge from the starting price. The savings was measured because it was the top-level goal given to the participants.

==== Communication ====
* ''Elaborative statements'': the number of elaboration, providing a justification, or providing an implication for a statement were counted and summed and divided by the total number of statements for each group to derive a ''elaborative'' score.
* ''Critical statements'': the number of counter-suggestions, clarification questions, requests for justification, and evaluation statements were counted and summed and divided by the total number of statements for each group to derive a ''critical'' score.

=== Findings ===
'''Learning'''

* For ''shallow'' measures of learning, that is, the [[normal post-test]] measures, there were no differences between conditions ''F'' (2, 133) < 1. See the text-explicit column in the table below.
* For ''deep'' measures of [[transfer]] of learning (see the inferential column), there was a main effect of condition, reflecting a higher score for the elaborative dyads (''M'' = 11.18, ''SD'' = 18.81) than the control dyads (''M'' = 3.31, ''SD'' = 15.33), ''F'' (2, 133) = 3.08, ''p'' < .05.

<tt>
{| border="1" cellpadding="5" cellspacing="0" align="center"
| rowspan="2"|
! colspan="2" | Text-explicit
! colspan="2" | Inferential
|-
| M
| SD
| M
| SD
|-
| Individuals||0.51||0.32||0.05||0.16
|-
| Control Dyads||0.59||0.27||0.03||0.19
|-
| Critical Dyads||0.52||0.23||0.02||0.20
|-
| Elaborative Dyads||0.53||0.41||0.12||0.21
|-
|}
</tt>

'''Problem solving'''

* There was a marginal effect of condition on optimization score, ''F'' (2, 75) = 2.60, ''p'' = .08. Post-hoc analyses revealed a reliable difference between the elaborative dyads and control dyads, ''d'' = .61, but no difference between the individuals.
* Elaborating a partner’s ideas and suggestions increased the dyads’ ability to optimize their designs.
* On the other hand, there was no effect of condition on savings, suggesting all conditions constructed equally priced bridges, ''F'' (2, 75) < 1.

<tt>
{| border="1" cellpadding="5" cellspacing="0" align="center"
| rowspan="2"|
! colspan="2" | Iterations
! colspan="2" | Savings
! colspan="2" | Optimization Score
|-
| M
| SD
| M
| SD
| M
| SD
|-
|Individuals||49.55||23.36||50283.47||23073.10||0.56||0.13
|-
|Control Dyads||39.65||8.73||49974.52||20718.92||0.57||0.11
|-
|Critical Dyads||33.80||16.66||42415.56||24537.62||0.53||0.11
|-
|Elaborative Dyads||41.00||18.62||55541.21||26621.19||0.63||0.15
|-
|}
</tt>

'''Communication'''

* The elaborative dyads asked fewer ''clarification questions'' than the control condition.
* The elaborative condition explicitly mentioned and used the ''simulation feedback'' more than the control condition.
* The elaborative dyads were marginally more likely to be classified as using a ''positive problem-solving strategy'' than the control dyads.

=== Explanation ===

Elaboration may facilitate the rapid assignment of variables to unfilled slots. The results suggest that elaboration may have been effective in filling unassigned variables because the elaborative dyads asked fewer clarification questions than the control condition. Additionally, the results indicate that the elaborative dyads were better able to use the feedback from the simulation in deciding where to implement the changes.

Combining these two results, making faster variable assignments and better use of the simulation’s feedback, an exchange between two dyads in the elaboration condition might be interpreted in the following way. One person suggests that they modify the member properties by switching solid members to hollow. The second person may elaborate the suggestion by looking at the feedback and making a recommendation. If the second person makes explicit how she made her recommendation, then the use of the feedback is now available to the dyad for future use. The finding that the elaborative dyads were more likely to be classified as using positive [[strategies]] supports this interpretation. If the dyad is able to cover more of the design space, they will more likely be exposed to more knowledge components thereby learning deeper knowledge. Furthermore, the partner in the group can serve as an additional source of knowledge components. Members of a dyad are exposed to the knowledge embedded in the simulation, as well as the information shared by the partner.

=== Further Information ===
==== Annotated bibliography ====
* Submitted to the ''Journal of Educational Psychology'' on March 19, 2008.
* Presented at the Festschrift for Lauren Resnick, May, 2005 [http://www.pitt.edu/AFShome/c/h/chi/public/html/TalkAndDialogue]
* Presentation to the NSF Site Visitors, May, 2005
* Presented at CogSci2006, July, 2006
* Some results were presented to the Intelligent Tutoring in Serious Games workshop, Aug. 2006 [http://projects.ict.usc.edu/itgs/talks/Hausmann_Generative%20Dialogue%20Patterns.ppt]

==== References ====
# Hausmann, R. G. M. (2006). Why do elaborative dialogs lead to effective problem solving and deep learning? In R. Sun & N. Miyake (Eds.), [http://www.learnlab.org/uploads/mypslc/publications/pos491-hausmann.pdf 28th Annual Meeting of the Cognitive Science Society] (pp. 1465-1469). Vancouver, B.C.: Sheridan Printing.
# Hausmann, R. G. M. (2005). [http://etd.library.pitt.edu/ETD/available/etd-08172005-144859 Elaborative and critical dialog: Two potentially effective problem-solving and learning interactions.] Unpublished Dissertation, University Of Pittsburgh, Pittsburgh, PA.

==== Connections ====
* [[Rummel Scripted Collaborative Problem Solving | Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving]]
* [[Using Elaborated Explanations to Support Geometry Learning (Aleven & Butcher) | Using Elaborated Explanations to Support Geometry Learning]]

==== Future plans ====
* Replicate with stronger manipulation of critical interactions.

File:IND.JPG

2008-10-10T16:08:33Z

Bobhaus:

File:Diss Control.JPG

2008-10-10T16:08:22Z

Bobhaus:

File:ICR.JPG

2008-10-10T16:07:59Z

Bobhaus:

File:IER.JPG

2008-10-10T16:07:44Z

Bobhaus:

File:Joint SE.JPG

2008-10-10T14:38:41Z

Bobhaus:

File:Individual SE.JPG

2008-10-10T14:38:33Z

Bobhaus:

Hausmann Study2

2008-10-10T14:38:21Z

Bobhaus: /* Independent variables */

== The Effects of Interaction on Robust Learning ==
''Robert Hausmann and Kurt VanLehn''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Robert G.M. Hausmann & Kurt VanLehn
|-
| '''Other Contributers''' || Robert N. Shelby (USNA), Brett van de Sande (Pitt)
|-
| '''Study Start Date''' || Sept. 1, 2006
|-
| '''Study End Date''' || Aug. 31, 2007
|-
| '''LearnLab Site''' || none (''in vitro'' study)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = 39
|-
| '''Total Participant Hours''' || 78 hrs.
|-
| '''DataShop''' || Loaded: 11/02/07
|}
 

=== Abstract ===
It is widely assumed that an interactive learning resource is more effective in producing learning gains than non-interactive [[sources]]. It turns out, however, that this assumption may not be completely accurate. For instance, research on human tutoring suggests that human tutoring (i.e., interactive) is just as effective as reading a textbook (i.e., non-interactive) under very particular circumstances (VanLehn et al., 2007). This raises the question, under which conditions should we expect to observe strong learning gains from interactive learning situations?

The current project seeks to address this question by contrasting interactive learning (i.e., jointly constructing explanations) with non-interactive learning (i.e., individually constructing explanations). Students were prompted to either self-explain in the singleton condition or to jointly construct explanations in the dyad condition.

=== Background and Significance ===
Several studies on collaborative learning have shown that it is more effective in producing learning gains than learning the same material alone. This finding has been replicated in many different configurations of students and across several different domains. Once the effect was established, the field moved into a more interesting phase, which was to accurately describe the interactions themselves and their impact on student learning (Dillenbourg, 1999). One of the hot topics in collaborative research is on the "co-construction" of new knowledge. Co-construction has been defined in many different ways. Therefore, the present study limits the scope of co-constructed ideas to [[jointly constructed explanation]]s.

Evidence supporting [[jointly constructed explanation]]s is sparse, but can be found in a study by McGregor and Chi (2002). They found that collaborative peers are able to not only jointly constructed ideas, but they will also reuse the ideas in a later problem-solving session. One of the limitations of their study was that it did not measure the impact of jointly constructed ideas on robust learning. In a related study, Hausmann, Chi, and Roy (2004) found correlational evidence for learning from co-construction. To provide more stringent evidence for the impact of jointly constructed explanations, the present study will manipulate the types of conversations dyads have by [[prompting]] for [[jointly constructed explanation]]s and measuring the effect on robust learning.

=== Glossary ===
* [[Jointly constructed explanation]]
* [[Prompting]]

See [[:Category:Hausmann_Study2|Hausmann_Study2 Glossary]]

=== Research question ===
How is [[robust learning]] affected by [[self-explanation]] vs. [[jointly constructed explanation]]s?

=== Independent variables ===
Only one independent variable, with two levels, was used:
* Explanation-construction: individually constructed explanations vs. jointly constructed explanations

[[Prompting]] for an explanation was intended to increase the probability that the individual or dyad will traverse a useful learning-event path.

Figure 1. An example from the Individual Self-explanation condition 
[[Image:Individual_SE.JPG]] 

Figure 2. An example from the Joint Self-explanation condition 
[[Image:Joint_SE.JPG]] 

=== Hypothesis ===
'''The Interactive Hypothesis''': collaborative peers will learn more than the individual learners because they benefit from the process of negotiating meaning with a peer, of appropriating part of the peers’ perspective, of building and maintaining common ground, and of articulating their knowledge and clarifying it when the peer misunderstands. In terms of the Interactive Communication cluster, the hypothesis states that, even when controlling for the amount of knowledge components covered, the dyads will learn more than the individuals.

This experiment implements the following instructional principle: [[Prompted Self-explanation]]

=== Dependent variables ===
* ''Near transfer, immediate'': electrodynamics problems solved in [[Andes]] during the laboratory period (2 hrs.).

=== Results ===
'''Laboratory Experiment'''
We conducted a laboratory (''in vitro'' not ''in vivo'') experiment during the Spring 2007 semester. Undergraduate volunteers, who were enrolled in the second semester of physics at the University of Pittsburgh, composed the sample for the study. Unfortunately, the sample size was small because our pool of participants was extremely limited.

'''Procedure'''
Participants were randomly assigned to condition. The first activity was to train the participants in their respective explanation activities. They read the instructions to the experiment, presented on a webpage, followed by the prompts used after each step of the example. Finally, they read an example of a hypothetical self-explanation or joint explanation.

After reading the experimental instructions, they then watched an introductory video to the Andes physics tutor. Afterwards, they solved a simple, single-principle electrodynamics problem (i.e., the warm-up problem). Once they finished, participants then watched a video solving an ''isomorphic problem''. Note that this procedure is slightly different from those used in the past where examples are presented before solving problems (e.g., Sweller & Cooper, 1985; Exper. 2). The videos decomposed into steps, and students were prompted to explain each step. The cycle of explaining examples and solving problems repeated until either 4 problems were solved or 2 hours elapsed. The first problem was used as a warm-up exercise, and the problems became progressively more complex.

As in our [[Hausmann_Study | first experiment]], we used normalize assistance scores. Normalize assistance scores were defined as the sum of all the errors and requests for help on that problem divided by the number of entries made in solving that problem. Thus, lower assistance scores indicate that the student derived a solution while making fewer mistakes and getting less help, and thus demonstrating better performance and understanding.

The results from the laboratory study were as follows:

* '''Differences between conditions'''

The jointly constructed explanation (JCE) condition (''M'' = .45, ''SD'' = .33) demonstrated lower assistance scores than the individually constructed explanation (ICE) condition (''M'' = 1.00, ''SD'' = .67). The difference between experimental conditions was statistically reliable and of high practical significance, ''F''(1, 23) = 7.33, ''p'' = .01, ηp2 = .24.

* '''Problem by condition'''

The pattern observed at the level of conditions replicated at the level of problem. That is, when the problem was used as a within factor in a repeated measures analysis of variance (MANOVA), the JCE condition demonstrated lower normalized assistance scores for all of the problems, except for the first, warm-up problem (see Table).

 
{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|
|ICE (n = 9)
|JCE (n = 14)
|''p''
|ηp2
|-
|Warm-up ||0.75 ||0.63 ||.483 ||.024
|-
|Problem 2 || 1.09 || 0.32 ||.003 ||.341
|-
|Problem 3 || 1.08 || 0.51 ||.059 ||.160
|-
|Problem 4 || 0.67 || 0.29 ||.034 ||.196
|}
 

In addition to providing higher quality solutions, the jointly constructed explanation condition (''M'' = 985.71, ''SD'' = 45.60) also solved their problems more quickly than the individually constructed explanation condition (''M'' = 1097.75, ''SD'' = 51.45). Although the omnibus difference between experimental conditions was not statistically reliable, ''F''(1, 23) = 2.66, ''p'' = .12, ηp2 = .10, the differences in solution times for the second and third problem were reliably lower for the JCE condition. This finding is particularly interesting because the experiment was capped at two hours; therefore, the dyads were able to complete the problem set more often than the individuals, χ2 = 4.81, ''p'' = .03. There were no reliable differences in terms of the amount of time explaining the steps of the worked-out examples.

* '''Knowledge component (KC) by condition'''

Because not all of the individuals were able to complete the entire problem set, their data could not be included in an analysis of the knowledge components. A MANOVA assumes that each individual participates in all of the measures. However, this is not the case when the individuals did not complete the last problem. Therefore, this fine-grained analysis of learning will need to wait until the study can be replicated in the classroom, with a larger sample size.

In an effort to anticipate the replication, we used a data imputation method for estimating missing values for each student (SPSS v14.0: Transform/Replace missing values/Median of nearby points). This provided a complete dataset for all participants. The results are summarized in the table below.

<tt>
{| border="1" cellspacing="0" cellpadding="5" align="center"
|-
|MAG || <center>[[Image:mag_learning_curves.JPG]]</center> || EFIELD || <center>[[Image:efield_learning_curves.JPG]]</center>
|-
|FORCE || <center>[[Image:force_learning_curves.JPG]]</center> ||CHARGE || <center>[[Image:charge_learning_curves.JPG]]</center>
|-
|}
</tt>
<center>'''Table 2'''. Learning curves for each knowledge component.</center>
 

=== Explanation ===
This study is part of the [[Interactive_Communication|Interactive Communication cluster]], and it hypothesizes that the [[prompting]] singletons and dyads should increase the probability that they traverse useful [[learning events]]. However, it is unclear if the act of communicating with a partner should increase learning, if the type of statements (i.e., explanations) are held constant. A strong-sense version of the Interactive Communication hypothesis would suggest that interacting with a peer is beneficial for learning because dyads can learn from their partner by assimilating [[knowledge component]]s articulated by the partner or via corrective comments that help to refine vague or incorrect [[knowledge component]]s.

Both [[self-explanation]] and joint-explanation should lead to deeper [[Knowledge Construction Dialogues]] or monolgues because they are likely to include integrative statements that connect information with prior knowledge, connect information with or previously stated material, or infer new knowledge. However, it may be more likely that deeper knowledge construction occurs during dialogue because the communicative partner provides a social cue to avoid glossing over the material.

=== Further Information ===
==== Annotated bibliography ====
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, July). Are self-explaining and coached problem solving more effective when done by pairs of students than alone? Poster presented at the 30th meeting of the Cognitive Science Society, Washington, DC.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Presentation to the PSLC Advisory Board: January, 2008
* Conference proceeding submitted to CogSci 2008
* Presented at a PSLC lunch: June 12, 2006 [http://www.learnlab.org/uploads/mypslc/talks/hausmannprojectproposalv4.ppt]
* Presented to the Physics LearnLab: April 4, 2007

==== References ====
# Dillenbourg, P. (1999). What do you mean "collaborative learning"? In P. Dillenbourg (Ed.), ''Collaborative learning: Cognitive and computational approaches'' (pp. 1-19). Oxford: Elsevier. [http://tecfa.unige.ch/tecfa/publicat/dil-papers-2/Dil.7.1.14.pdf]
# Hausmann, R. G. M., & Chi, M. T. H. (2002). Can a computer interface support self-explaining? ''Cognitive Technology, 7''(1), 4-14. [http://www.pitt.edu/~bobhaus/hausmann2002.pdf]
# Hausmann, R. G. M., Chi, M. T. H., & Roy, M. (2004). Learning from collaborative problem solving: An analysis of three hypothesized mechanisms. In K. D. Forbus, D. Gentner & T. Regier (Eds.), ''26nd Annual Conference of the Cognitive Science Society'' (pp. 547-552). Mahwah, NJ: Lawrence Erlbaum. [http://www.cogsci.northwestern.edu/cogsci2004/papers/paper445.pdf]
# McGregor, M., & Chi, M. T. H. (2002). Collaborative interactions: The process of joint production and individual reuse of novel ideas. In W. D. Gray & C. D. Schunn (Eds.), ''24nd Annual Conference of the Cognitive Science Society.'' Mahwah, NJ: Lawerence Erlbaum. [http://www.cognitivesciencesociety.org/confproc/gmu02/final_ind_files/McGregor_chi.pdf]
# Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. ''Cognition and Instruction, 2''(1), 59-89. [http://web.ebscohost.com/ehost/pdf?vid=2&hid=105&sid=34dbdad3-7874-419b-aace-3a9b56a3f41d%40sessionmgr107]
# VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are tutorial dialogues more effective than reading? ''Cognitive Science. 31''(1), 3-62. [http://www.leaonline.com/doi/pdfplus/10.1207/s15516709cog3101_2]

==== Connections ====
* [[Craig observing | Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi)]]

==== Future plans ====
Our future plans for June 2007 - August 2007:
* Transcribe the interactions and code them for explanations, either constructed jointly or individually.

Hausmann Study

2008-10-10T14:35:25Z

Bobhaus: /* Independent variables */

== A comparison of self-explanation to instructional explanation ==
''Robert Hausmann and Kurt VanLehn''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Robert G.M. Hausmann & Kurt VanLehn
|-
| '''Other Contributers''' * '''Faculty: ''' * '''Staff: '''
| Donald J. Treacy (USNA), Robert N. Shelby (USNA) Brett van de Sande (Pitt), Anders Weinstein (Pitt)
|-
| '''Study Start Date''' || Sept. 1, 2005
|-
| '''Study End Date''' || Aug. 31, 2006
|-
| '''LearnLab Site''' || USNA Physics II
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = 104
|-
| '''Total Participant Hours''' || 190 hrs.
|-
| '''DataShop''' || Loaded: 10/11/2007
|}
 

=== Abstract ===
This [[in vivo experiment]] compared the learning that results from either hearing an explanation ([[instructional explanation]]) or generating it oneself ([[self-explanation]]). Students studied a physics [[example]]s, which were presented step by step. For half the students, the [[example]] steps were incompletely justified (i.e., the explanations connecting some steps were missing) whereas for the other half, the [[example]] steps were completely justified. Crossed with this variable was an attempted manipulation of the student’s studying strategy. Half the students were instructed and prompted to self-explain at the end of each step, while the other half were prompted to paraphrase it (click the following [http://andes3.lrdc.pitt.edu/~bob/mat/cond.html link] for screen shots of each experimental condition). Paraphrasing was selected as the contrasting study strategy because earlier work has shown that paraphrasing suppresses [[self-explanation]].

Of the four conditions, one condition was key to testing our hypothesis: the condition where students viewed completely justified [[example]]s and were asked to paraphrase them. If the learning gains from this condition had been just as high as those from conditions were self-explanation was encouraged, then instructional explanation would have been just as effective as self-explanation. On the other hand, if the learning gains were just as low as those where instructional explanations were absent and self-explanation was suppressed via paraphrasing, then instructional explanation would have been just as ineffective as no explanation at all.

Preliminary results of the normal and robust learning measures suggest that the latter case occurred, so instructional explanation were not as effective as self-explanations.

=== Background and Significance ===

The first studies of self-explanation, which were based on analysis of verbal protocols, showed that the amount of self-explaining correlated strongly with performance on post-test measures of problem-solving performance (Bielaczyc, Pirolli, & Brown, 1995; Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Renkl, 1997). Because these studies compared self-explanation to the lack of any explanation at all, there is a confound between the generative act or producing the explanations and the additional content of the explanations themselves. Perhaps if students were simply given these explanations, they would learn just as much. Alternatively, learning from self-explaining might arise from the activity of producing the explanations. Thus, if they were given the explanations, they would not learn just as much. In other words, is it merely attending to the explanations that matters, or is robust learning more likely to occur if students generate the explanations themselves? Let us label these hypotheses as follows:

# '''The Attention Hypothesis''': learning from self-generated explanations should produce comparable learning gains as author-provided explanation, provided the learner pays attention to them. Both self-generated and author-provided explanations should exhibit better learning than no explanation.
# '''The Generation Hypothesis''': learning from self-generated explanations should produce greater learning gains than author-provided explanations because they are produced from the students’ own background knowledge; however, author-provided explanations should be comparable to no explanation.

There have only been a few empirical studies that attempt to separate the Attention hypothesis from the Generation hypothesis (Brown & Kane, 1988; Schworm & Renkl, 2002). An exemplary case can be found in a study by Lovett (1992) in the domain of permutation and combination problems. Lovett crossed the source of the solution (subject vs. experimenter) with the source of the explanation for the solution (subject vs. experimenter). For our purposes, only two of the experimental conditions matter. The experimenter-subject condition was analogous to experimental materials found in a self-explanation experiment wherein the students self-explained an author’s solution, whereas in the experimenter-experimenter condition, the students studied an author-provided explanation. Lovett found that the experimenter-experimenter condition demonstrated better performance, especially on far-transfer items. Lovett’s interpretation was that the experimenter-experimenter condition was effective because it contained higher quality explanations than those generated by students. Consistent with this interpretation, when Lovett analyzed the protocol data, she found that the participants who generated the key inferences had the same learning gains as participants who read the corresponding inferences. Thus, of our two hypotheses, Lovett’s experiment supports the Attention hypothesis: the content of self-explanations matters, while the source of the explanation does not.

Brown and Kane (1988) found that children’s explanations, generated either spontaneously or in response to prompting, were much more effective at promoting transfer than those provided by the experimenter. In particular, students were first told a story about mimicry. Some students were then told, "Some animals try to look like a scary animal so they won’t get eaten.” Other students were asked first, “Why would a furry caterpillar want to look like a snake?” and if that did not elicit an explanation, they were asked, "What could the furry caterpillar do to stop the big birds from eating him?" Most students got the question right, and if they did, 85% were able to answer a similar question about two new stories. If they were told the rule, then only 45% were able to answer a similar question about the new stories. This result is consistent with the Generation hypothesis, which is that an explanation is effective when the student generates it. However, the students who were told the rule may not have paid much attention to it, according to Brown and Kane.

In summary, one study’s results are consistent with the Attention hypothesis, and the other study’s results are consistent with the Generation hypothesis, but both studies confounded two variables. In the Lovett study, the student-produced and author-provided explanations were of different qualities. In the Brown and Kane study, the students in the author-provided explanations condition may not have paid much attention to the explanations.

=== Glossary ===
* [[Physics example line]]
* [[Complete vs. incomplete example]]
* [[Instructional explanation]]
See [[:Category:Hausmann_Study|Hausmann_Study Glossary]]

=== Research question ===
How is robust learning affected by [[self-explanation]] vs. [[instructional explanation]]?

=== Independent variables ===
Two variables were crossed:
* Did the [[example]] present an explanation with each step or present just the step?
* After each step (and its explanation, if any) was presented, students were prompted to either further explain the step or paraphrase the step in their own words.

The condition where explanations were presented in the [[example]] and students were asked to paraphrase them is considered the “instructional explanation” condition. The two conditions where students were asked to self-explain the [[example]] lines are considered the “self-explanation” conditions. The remaining condition, where students were asked to paraphrase [[example]]s that did not contain explanations, was considered the “no explanation” condition. Note, however, that nothing prevents a student in the paraphrasing condition to self-explain (and vice-versa).

Figure 1. An example from the Complete Self-explanation condition 
[[Image:CSE.JPG]] 

Figure 2. An example from the Incomplete Self-explanation condition 
[[Image:ISE.JPG]] 

Figure 3. An example from the Complete Paraphrase condition 
[[Image:CPP.JPG]] 

Figure 4. An example from the Incomplete Paraphrase condition 
[[Image:IPP.JPG]] 

=== Hypothesis ===
For these well-prepared students, self-explanation should not be too difficult. That is, the instruction should be below the students’ zone of proximal development. Thus, the learning-by-doing path (self-explanation) should elicit more robust learning than the alternative path (instructional explanation) wherein the student does less work.

As a manipulation check on the utility of the explanations in the complete [[example]]s, we hypothesize that the instructional explanation condition should produce more robust learning than the no-explanation condition.

=== Dependent variables ===
* [[Normal post-test]]
** ''Near transfer, immediate'': During training, [[worked examples]] alternated with problems, and the problems were solved using [[Andes]]. Each problem was similar to the [[worked examples|example]] that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). The log data were analyzed and assistance scores (sum of errors and help requests, normalized by the number of transactions) were calculated.

* [[Robust learning]]
** ''[[Long-term retention]]'': On the student’s regular mid-term exam, one problem was similar to the training. Since this exam occurred a week after the training, and the training took place in just under 2 hours, the student’s performance on this problem is considered a test of [[long-term retention]].
** ''Near and far [[transfer]]'': After training, students did their regular homework problems using [[Andes]]. Students did them whenever they wanted, but most completed them just before the exam. The homework problems were divided based on similarity to the training problems, and assistance scores were calculated.
** ''[[Accelerated future learning]]'': The training was on electrical fields, and it was followed in the course by a unit on magnetic fields. Log data from the magnetic field homework was analyzed as a measure of acceleration of future learning.

=== Results ===
* [[Normal post-test]]
** ''Near [[transfer]], immediate'': The self-explanation condition demonstrating lower normalized assistance scores than the paraphrase condition, ''F''(1, 73) = 6.19, ''p'' = .02, ηp2 =.08. <center>[[Image:NTI_results.JPG]]</center>

* [[Robust learning]]
** ''Near transfer, retention'': Results on the measure were mixed. While there were no reliable main effects or interactions, the complete self-explanation group was marginally higher than the complete paraphrase condition (LSD, p = .06). Moreover, we analyzed the students’ performance on a homework problem that was isomorphic to the chapter exam in that they shared an identical deep structure (i.e., both analyzed the motion of a charged particle moving in two dimensions). The self-explanation had reliably lower normalized assistance scores than the paraphrase condition, ''F''(1, 27) = 4.07, ''p'' = .05, ηp2 = .13. <center>[[Image:FTR_results.JPG]]</center>
** ''Near and far transfer'':
** ''[[Accelerated future learning]]'': There was no effect for [[example]] completeness; however, the self-explanation condition demonstrating lower normalized assistance scores than the paraphrase condition, ''F''(1, 46) = 5.22, ''p'' = .03, ηp2 = .10. <center>[[Image:AFL_results.JPG]]</center>

=== Explanation ===
This study is part of the [[Interactive_Communication|Interactive Communication cluster]], and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s hypothesis is that robust learning occurs when two conditions are met:

* The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work. In this study, self-explanation comprises the learning-by-doing path and instructional explanations are ones where another agent (the author of the text) has done most of the work.

* The student takes the learning-by-doing path unless it becomes too difficult. This study tried (successfully, it appears) to control the student’s path choice. It showed that when students take the learning-by-doing path, they learned more than when they took the alternative path.

The IC cluster’s hypothesis actually predicts an attribute-treatment interaction (ATI) here. If some students were under-prepared and thus would find the self-explanation path too difficult, then those students would learn more on the instructional-explanation path. ATI analyzes have not yet been completed.

=== Further Information ===
==== Annotated bibliography ====
* Accepted as a poster at CogSci 2008
* Presentation to the Higher-order Cognition Group, led by Timothy J. Nokes, on March 3, 2008
* Presentation to the PSLC Advisory Board, December, 2006 [http://www.learnlab.org/uploads/mypslc/talks/hausmannabvisit2006bv2.ppt]
* Presentation to the NSF Follow-up Site Visitors, September, 2006
* Preliminary results were presented to the Intelligent Tutoring in Serious Games workshop, Aug. 2006 [http://projects.ict.usc.edu/itgs/talks/Hausmann_Generative%20Dialogue%20Patterns.ppt]
* Presentation to the NSF Site Visitors, June, 2006
* Poster presented at the annual meeting of the Science of Learning Centers, Oct. 2006.
* Symposium accepted to EARLI 2007
* Symposium accepted at AERA 2007
* Full-paper accepted at AIED 2007 [http://www.learnlab.org/uploads/mypslc/publications/hausmannvanlehn2007_final.pdf]
* Accepted as a poster at CogSci 2007
* Presentation to the NSF Site Visitors, May 31, 2007

==== References ====
# Anzai, Y., & Simon, H. A. (1979). The theory of learning by doing. ''Psychological Review, 86''(2), 124-140.
# Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. ''Cognitive Science, 13'', 145-182. [http://rt4rf9qn2y.scholar.serialssolutions.com/?sid=google&auinit=MTH&aulast=CHI&atitle=Self-explanations:+how+students+study+and+use+examples+in+learning+to+solve+problems&title=Cognitive+science&volume=13&issue=2&date=1989&spage=145&issn=0364-0213]
# Hausmann, R. G. M., & Chi, M. T. H. (2002). Can a computer interface support self-explaining? ''Cognitive Technology, 7''(1), 4-14. [http://www.pitt.edu/~bobhaus/hausmann2002.pdf]
# Lovett, M. C. (1992). Learning by problem solving versus by examples: The benefits of generating and receiving information. ''Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society'' (pp. 956-961). Hillsdale, NJ: Erlbaum.
# Schworm, S., & Renkl, A. (2002). Learning by solved example problems: Instructional explanations reduce self-explanation activity. In W. D. Gray & C. D. Schunn (Eds.), ''Proceedings of the 24th Annual Conference of the Cognitive Science Society'' (pp. 816-821). Mahwah, NJ: Erlbaum.[http://www.cognitivesciencesociety.org/confproc/gmu02/final_ind_files/schworm_renkl.pdf]
# Schworm, S., & Renkl, A. (2006). Computer-supported example-based learning: When instructional explanations reduce self-explanations. ''Computers & Education, 46,'' 426-445.

==== Connections ====
This project shares features with the following research projects:

* [[Booth | Knowledge component construction vs. recall (Booth, Siegler, Koedinger & Rittle-Johnson)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & VanLehn)]]
* [[Craig observing | Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi)]]

==== Future plans ====
Our future plans for June 2007 - August 2007:
* Code transcripts for explanations and paraphrases.
* Link codes from transcript to knowledge components.
* Link transcript to steps in Andes log files.
* Submit to the International Journal of Artificial Intelligence in Education

Hausmann Study

2008-10-10T14:34:41Z

Bobhaus: /* Independent variables */

== A comparison of self-explanation to instructional explanation ==
''Robert Hausmann and Kurt VanLehn''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Robert G.M. Hausmann & Kurt VanLehn
|-
| '''Other Contributers''' * '''Faculty: ''' * '''Staff: '''
| Donald J. Treacy (USNA), Robert N. Shelby (USNA) Brett van de Sande (Pitt), Anders Weinstein (Pitt)
|-
| '''Study Start Date''' || Sept. 1, 2005
|-
| '''Study End Date''' || Aug. 31, 2006
|-
| '''LearnLab Site''' || USNA Physics II
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = 104
|-
| '''Total Participant Hours''' || 190 hrs.
|-
| '''DataShop''' || Loaded: 10/11/2007
|}
 

=== Abstract ===
This [[in vivo experiment]] compared the learning that results from either hearing an explanation ([[instructional explanation]]) or generating it oneself ([[self-explanation]]). Students studied a physics [[example]]s, which were presented step by step. For half the students, the [[example]] steps were incompletely justified (i.e., the explanations connecting some steps were missing) whereas for the other half, the [[example]] steps were completely justified. Crossed with this variable was an attempted manipulation of the student’s studying strategy. Half the students were instructed and prompted to self-explain at the end of each step, while the other half were prompted to paraphrase it (click the following [http://andes3.lrdc.pitt.edu/~bob/mat/cond.html link] for screen shots of each experimental condition). Paraphrasing was selected as the contrasting study strategy because earlier work has shown that paraphrasing suppresses [[self-explanation]].

Of the four conditions, one condition was key to testing our hypothesis: the condition where students viewed completely justified [[example]]s and were asked to paraphrase them. If the learning gains from this condition had been just as high as those from conditions were self-explanation was encouraged, then instructional explanation would have been just as effective as self-explanation. On the other hand, if the learning gains were just as low as those where instructional explanations were absent and self-explanation was suppressed via paraphrasing, then instructional explanation would have been just as ineffective as no explanation at all.

Preliminary results of the normal and robust learning measures suggest that the latter case occurred, so instructional explanation were not as effective as self-explanations.

=== Background and Significance ===

The first studies of self-explanation, which were based on analysis of verbal protocols, showed that the amount of self-explaining correlated strongly with performance on post-test measures of problem-solving performance (Bielaczyc, Pirolli, & Brown, 1995; Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Renkl, 1997). Because these studies compared self-explanation to the lack of any explanation at all, there is a confound between the generative act or producing the explanations and the additional content of the explanations themselves. Perhaps if students were simply given these explanations, they would learn just as much. Alternatively, learning from self-explaining might arise from the activity of producing the explanations. Thus, if they were given the explanations, they would not learn just as much. In other words, is it merely attending to the explanations that matters, or is robust learning more likely to occur if students generate the explanations themselves? Let us label these hypotheses as follows:

# '''The Attention Hypothesis''': learning from self-generated explanations should produce comparable learning gains as author-provided explanation, provided the learner pays attention to them. Both self-generated and author-provided explanations should exhibit better learning than no explanation.
# '''The Generation Hypothesis''': learning from self-generated explanations should produce greater learning gains than author-provided explanations because they are produced from the students’ own background knowledge; however, author-provided explanations should be comparable to no explanation.

There have only been a few empirical studies that attempt to separate the Attention hypothesis from the Generation hypothesis (Brown & Kane, 1988; Schworm & Renkl, 2002). An exemplary case can be found in a study by Lovett (1992) in the domain of permutation and combination problems. Lovett crossed the source of the solution (subject vs. experimenter) with the source of the explanation for the solution (subject vs. experimenter). For our purposes, only two of the experimental conditions matter. The experimenter-subject condition was analogous to experimental materials found in a self-explanation experiment wherein the students self-explained an author’s solution, whereas in the experimenter-experimenter condition, the students studied an author-provided explanation. Lovett found that the experimenter-experimenter condition demonstrated better performance, especially on far-transfer items. Lovett’s interpretation was that the experimenter-experimenter condition was effective because it contained higher quality explanations than those generated by students. Consistent with this interpretation, when Lovett analyzed the protocol data, she found that the participants who generated the key inferences had the same learning gains as participants who read the corresponding inferences. Thus, of our two hypotheses, Lovett’s experiment supports the Attention hypothesis: the content of self-explanations matters, while the source of the explanation does not.

Brown and Kane (1988) found that children’s explanations, generated either spontaneously or in response to prompting, were much more effective at promoting transfer than those provided by the experimenter. In particular, students were first told a story about mimicry. Some students were then told, "Some animals try to look like a scary animal so they won’t get eaten.” Other students were asked first, “Why would a furry caterpillar want to look like a snake?” and if that did not elicit an explanation, they were asked, "What could the furry caterpillar do to stop the big birds from eating him?" Most students got the question right, and if they did, 85% were able to answer a similar question about two new stories. If they were told the rule, then only 45% were able to answer a similar question about the new stories. This result is consistent with the Generation hypothesis, which is that an explanation is effective when the student generates it. However, the students who were told the rule may not have paid much attention to it, according to Brown and Kane.

In summary, one study’s results are consistent with the Attention hypothesis, and the other study’s results are consistent with the Generation hypothesis, but both studies confounded two variables. In the Lovett study, the student-produced and author-provided explanations were of different qualities. In the Brown and Kane study, the students in the author-provided explanations condition may not have paid much attention to the explanations.

=== Glossary ===
* [[Physics example line]]
* [[Complete vs. incomplete example]]
* [[Instructional explanation]]
See [[:Category:Hausmann_Study|Hausmann_Study Glossary]]

=== Research question ===
How is robust learning affected by [[self-explanation]] vs. [[instructional explanation]]?

=== Independent variables ===
Two variables were crossed:
* Did the [[example]] present an explanation with each step or present just the step?
* After each step (and its explanation, if any) was presented, students were prompted to either further explain the step or paraphrase the step in their own words.

The condition where explanations were presented in the [[example]] and students were asked to paraphrase them is considered the “instructional explanation” condition. The two conditions where students were asked to self-explain the [[example]] lines are considered the “self-explanation” conditions. The remaining condition, where students were asked to paraphrase [[example]]s that did not contain explanations, was considered the “no explanation” condition. Note, however, that nothing prevents a student in the paraphrasing condition to self-explain (and vice-versa).

Figure 1. An example from the Complete Self-explanation condition 
[[Image:CSE.JPG]] 

Figure 2. An example from the Incomplete Self-explanation condition 
[[Image:ISE.JPG]] 

Figure 3. An example from the Complete Paraphrase condition 
[[Image:CPP.JPG]] 

Figure 4. An example from the Incomplete Paraphrase condition 
[[Image:IPP.JPG]] 

=== Hypothesis ===
For these well-prepared students, self-explanation should not be too difficult. That is, the instruction should be below the students’ zone of proximal development. Thus, the learning-by-doing path (self-explanation) should elicit more robust learning than the alternative path (instructional explanation) wherein the student does less work.

As a manipulation check on the utility of the explanations in the complete [[example]]s, we hypothesize that the instructional explanation condition should produce more robust learning than the no-explanation condition.

=== Dependent variables ===
* [[Normal post-test]]
** ''Near transfer, immediate'': During training, [[worked examples]] alternated with problems, and the problems were solved using [[Andes]]. Each problem was similar to the [[worked examples|example]] that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). The log data were analyzed and assistance scores (sum of errors and help requests, normalized by the number of transactions) were calculated.

* [[Robust learning]]
** ''[[Long-term retention]]'': On the student’s regular mid-term exam, one problem was similar to the training. Since this exam occurred a week after the training, and the training took place in just under 2 hours, the student’s performance on this problem is considered a test of [[long-term retention]].
** ''Near and far [[transfer]]'': After training, students did their regular homework problems using [[Andes]]. Students did them whenever they wanted, but most completed them just before the exam. The homework problems were divided based on similarity to the training problems, and assistance scores were calculated.
** ''[[Accelerated future learning]]'': The training was on electrical fields, and it was followed in the course by a unit on magnetic fields. Log data from the magnetic field homework was analyzed as a measure of acceleration of future learning.

=== Results ===
* [[Normal post-test]]
** ''Near [[transfer]], immediate'': The self-explanation condition demonstrating lower normalized assistance scores than the paraphrase condition, ''F''(1, 73) = 6.19, ''p'' = .02, ηp2 =.08. <center>[[Image:NTI_results.JPG]]</center>

* [[Robust learning]]
** ''Near transfer, retention'': Results on the measure were mixed. While there were no reliable main effects or interactions, the complete self-explanation group was marginally higher than the complete paraphrase condition (LSD, p = .06). Moreover, we analyzed the students’ performance on a homework problem that was isomorphic to the chapter exam in that they shared an identical deep structure (i.e., both analyzed the motion of a charged particle moving in two dimensions). The self-explanation had reliably lower normalized assistance scores than the paraphrase condition, ''F''(1, 27) = 4.07, ''p'' = .05, ηp2 = .13. <center>[[Image:FTR_results.JPG]]</center>
** ''Near and far transfer'':
** ''[[Accelerated future learning]]'': There was no effect for [[example]] completeness; however, the self-explanation condition demonstrating lower normalized assistance scores than the paraphrase condition, ''F''(1, 46) = 5.22, ''p'' = .03, ηp2 = .10. <center>[[Image:AFL_results.JPG]]</center>

=== Explanation ===
This study is part of the [[Interactive_Communication|Interactive Communication cluster]], and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s hypothesis is that robust learning occurs when two conditions are met:

* The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work. In this study, self-explanation comprises the learning-by-doing path and instructional explanations are ones where another agent (the author of the text) has done most of the work.

* The student takes the learning-by-doing path unless it becomes too difficult. This study tried (successfully, it appears) to control the student’s path choice. It showed that when students take the learning-by-doing path, they learned more than when they took the alternative path.

The IC cluster’s hypothesis actually predicts an attribute-treatment interaction (ATI) here. If some students were under-prepared and thus would find the self-explanation path too difficult, then those students would learn more on the instructional-explanation path. ATI analyzes have not yet been completed.

=== Further Information ===
==== Annotated bibliography ====
* Accepted as a poster at CogSci 2008
* Presentation to the Higher-order Cognition Group, led by Timothy J. Nokes, on March 3, 2008
* Presentation to the PSLC Advisory Board, December, 2006 [http://www.learnlab.org/uploads/mypslc/talks/hausmannabvisit2006bv2.ppt]
* Presentation to the NSF Follow-up Site Visitors, September, 2006
* Preliminary results were presented to the Intelligent Tutoring in Serious Games workshop, Aug. 2006 [http://projects.ict.usc.edu/itgs/talks/Hausmann_Generative%20Dialogue%20Patterns.ppt]
* Presentation to the NSF Site Visitors, June, 2006
* Poster presented at the annual meeting of the Science of Learning Centers, Oct. 2006.
* Symposium accepted to EARLI 2007
* Symposium accepted at AERA 2007
* Full-paper accepted at AIED 2007 [http://www.learnlab.org/uploads/mypslc/publications/hausmannvanlehn2007_final.pdf]
* Accepted as a poster at CogSci 2007
* Presentation to the NSF Site Visitors, May 31, 2007

==== References ====
# Anzai, Y., & Simon, H. A. (1979). The theory of learning by doing. ''Psychological Review, 86''(2), 124-140.
# Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. ''Cognitive Science, 13'', 145-182. [http://rt4rf9qn2y.scholar.serialssolutions.com/?sid=google&auinit=MTH&aulast=CHI&atitle=Self-explanations:+how+students+study+and+use+examples+in+learning+to+solve+problems&title=Cognitive+science&volume=13&issue=2&date=1989&spage=145&issn=0364-0213]
# Hausmann, R. G. M., & Chi, M. T. H. (2002). Can a computer interface support self-explaining? ''Cognitive Technology, 7''(1), 4-14. [http://www.pitt.edu/~bobhaus/hausmann2002.pdf]
# Lovett, M. C. (1992). Learning by problem solving versus by examples: The benefits of generating and receiving information. ''Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society'' (pp. 956-961). Hillsdale, NJ: Erlbaum.
# Schworm, S., & Renkl, A. (2002). Learning by solved example problems: Instructional explanations reduce self-explanation activity. In W. D. Gray & C. D. Schunn (Eds.), ''Proceedings of the 24th Annual Conference of the Cognitive Science Society'' (pp. 816-821). Mahwah, NJ: Erlbaum.[http://www.cognitivesciencesociety.org/confproc/gmu02/final_ind_files/schworm_renkl.pdf]
# Schworm, S., & Renkl, A. (2006). Computer-supported example-based learning: When instructional explanations reduce self-explanations. ''Computers & Education, 46,'' 426-445.

==== Connections ====
This project shares features with the following research projects:

* [[Booth | Knowledge component construction vs. recall (Booth, Siegler, Koedinger & Rittle-Johnson)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & VanLehn)]]
* [[Craig observing | Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi)]]

==== Future plans ====
Our future plans for June 2007 - August 2007:
* Code transcripts for explanations and paraphrases.
* Link codes from transcript to knowledge components.
* Link transcript to steps in Andes log files.
* Submit to the International Journal of Artificial Intelligence in Education

Hausmann Study

2008-10-10T14:32:45Z

Bobhaus: /* Independent variables */

== A comparison of self-explanation to instructional explanation ==
''Robert Hausmann and Kurt VanLehn''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Robert G.M. Hausmann & Kurt VanLehn
|-
| '''Other Contributers''' * '''Faculty: ''' * '''Staff: '''
| Donald J. Treacy (USNA), Robert N. Shelby (USNA) Brett van de Sande (Pitt), Anders Weinstein (Pitt)
|-
| '''Study Start Date''' || Sept. 1, 2005
|-
| '''Study End Date''' || Aug. 31, 2006
|-
| '''LearnLab Site''' || USNA Physics II
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = 104
|-
| '''Total Participant Hours''' || 190 hrs.
|-
| '''DataShop''' || Loaded: 10/11/2007
|}
 

=== Abstract ===
This [[in vivo experiment]] compared the learning that results from either hearing an explanation ([[instructional explanation]]) or generating it oneself ([[self-explanation]]). Students studied a physics [[example]]s, which were presented step by step. For half the students, the [[example]] steps were incompletely justified (i.e., the explanations connecting some steps were missing) whereas for the other half, the [[example]] steps were completely justified. Crossed with this variable was an attempted manipulation of the student’s studying strategy. Half the students were instructed and prompted to self-explain at the end of each step, while the other half were prompted to paraphrase it (click the following [http://andes3.lrdc.pitt.edu/~bob/mat/cond.html link] for screen shots of each experimental condition). Paraphrasing was selected as the contrasting study strategy because earlier work has shown that paraphrasing suppresses [[self-explanation]].

Of the four conditions, one condition was key to testing our hypothesis: the condition where students viewed completely justified [[example]]s and were asked to paraphrase them. If the learning gains from this condition had been just as high as those from conditions were self-explanation was encouraged, then instructional explanation would have been just as effective as self-explanation. On the other hand, if the learning gains were just as low as those where instructional explanations were absent and self-explanation was suppressed via paraphrasing, then instructional explanation would have been just as ineffective as no explanation at all.

Preliminary results of the normal and robust learning measures suggest that the latter case occurred, so instructional explanation were not as effective as self-explanations.

=== Background and Significance ===

The first studies of self-explanation, which were based on analysis of verbal protocols, showed that the amount of self-explaining correlated strongly with performance on post-test measures of problem-solving performance (Bielaczyc, Pirolli, & Brown, 1995; Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Renkl, 1997). Because these studies compared self-explanation to the lack of any explanation at all, there is a confound between the generative act or producing the explanations and the additional content of the explanations themselves. Perhaps if students were simply given these explanations, they would learn just as much. Alternatively, learning from self-explaining might arise from the activity of producing the explanations. Thus, if they were given the explanations, they would not learn just as much. In other words, is it merely attending to the explanations that matters, or is robust learning more likely to occur if students generate the explanations themselves? Let us label these hypotheses as follows:

# '''The Attention Hypothesis''': learning from self-generated explanations should produce comparable learning gains as author-provided explanation, provided the learner pays attention to them. Both self-generated and author-provided explanations should exhibit better learning than no explanation.
# '''The Generation Hypothesis''': learning from self-generated explanations should produce greater learning gains than author-provided explanations because they are produced from the students’ own background knowledge; however, author-provided explanations should be comparable to no explanation.

There have only been a few empirical studies that attempt to separate the Attention hypothesis from the Generation hypothesis (Brown & Kane, 1988; Schworm & Renkl, 2002). An exemplary case can be found in a study by Lovett (1992) in the domain of permutation and combination problems. Lovett crossed the source of the solution (subject vs. experimenter) with the source of the explanation for the solution (subject vs. experimenter). For our purposes, only two of the experimental conditions matter. The experimenter-subject condition was analogous to experimental materials found in a self-explanation experiment wherein the students self-explained an author’s solution, whereas in the experimenter-experimenter condition, the students studied an author-provided explanation. Lovett found that the experimenter-experimenter condition demonstrated better performance, especially on far-transfer items. Lovett’s interpretation was that the experimenter-experimenter condition was effective because it contained higher quality explanations than those generated by students. Consistent with this interpretation, when Lovett analyzed the protocol data, she found that the participants who generated the key inferences had the same learning gains as participants who read the corresponding inferences. Thus, of our two hypotheses, Lovett’s experiment supports the Attention hypothesis: the content of self-explanations matters, while the source of the explanation does not.

Brown and Kane (1988) found that children’s explanations, generated either spontaneously or in response to prompting, were much more effective at promoting transfer than those provided by the experimenter. In particular, students were first told a story about mimicry. Some students were then told, "Some animals try to look like a scary animal so they won’t get eaten.” Other students were asked first, “Why would a furry caterpillar want to look like a snake?” and if that did not elicit an explanation, they were asked, "What could the furry caterpillar do to stop the big birds from eating him?" Most students got the question right, and if they did, 85% were able to answer a similar question about two new stories. If they were told the rule, then only 45% were able to answer a similar question about the new stories. This result is consistent with the Generation hypothesis, which is that an explanation is effective when the student generates it. However, the students who were told the rule may not have paid much attention to it, according to Brown and Kane.

In summary, one study’s results are consistent with the Attention hypothesis, and the other study’s results are consistent with the Generation hypothesis, but both studies confounded two variables. In the Lovett study, the student-produced and author-provided explanations were of different qualities. In the Brown and Kane study, the students in the author-provided explanations condition may not have paid much attention to the explanations.

=== Glossary ===
* [[Physics example line]]
* [[Complete vs. incomplete example]]
* [[Instructional explanation]]
See [[:Category:Hausmann_Study|Hausmann_Study Glossary]]

=== Research question ===
How is robust learning affected by [[self-explanation]] vs. [[instructional explanation]]?

=== Independent variables ===
Two variables were crossed:
* Did the [[example]] present an explanation with each step or present just the step?
* After each step (and its explanation, if any) was presented, students were prompted to either further explain the step or paraphrase the step in their own words.

The condition where explanations were presented in the [[example]] and students were asked to paraphrase them is considered the “instructional explanation” condition. The two conditions where students were asked to self-explain the [[example]] lines are considered the “self-explanation” conditions. The remaining condition, where students were asked to paraphrase [[example]]s that did not contain explanations, was considered the “no explanation” condition. Note, however, that nothing prevents a student in the paraphrasing condition to self-explain (and vice-versa).

Figure 1. An example from the Complete Self-explanation condition
[[Image:CSE.jpg]]

Figure 2. An example from the Incomplete Self-explanation condition
[[Image:ISE.jpg]]

Figure 3. An example from the Complete Paraphrase condition
[[Image:CPP.jpg]]

Figure 4. An example from the Incomplete Paraphrase condition
[[Image:IPP.jpg]]

=== Hypothesis ===
For these well-prepared students, self-explanation should not be too difficult. That is, the instruction should be below the students’ zone of proximal development. Thus, the learning-by-doing path (self-explanation) should elicit more robust learning than the alternative path (instructional explanation) wherein the student does less work.

As a manipulation check on the utility of the explanations in the complete [[example]]s, we hypothesize that the instructional explanation condition should produce more robust learning than the no-explanation condition.

=== Dependent variables ===
* [[Normal post-test]]
** ''Near transfer, immediate'': During training, [[worked examples]] alternated with problems, and the problems were solved using [[Andes]]. Each problem was similar to the [[worked examples|example]] that preceded it, so performance on it is a measure of normal learning (near transfer, immediate testing). The log data were analyzed and assistance scores (sum of errors and help requests, normalized by the number of transactions) were calculated.

* [[Robust learning]]
** ''[[Long-term retention]]'': On the student’s regular mid-term exam, one problem was similar to the training. Since this exam occurred a week after the training, and the training took place in just under 2 hours, the student’s performance on this problem is considered a test of [[long-term retention]].
** ''Near and far [[transfer]]'': After training, students did their regular homework problems using [[Andes]]. Students did them whenever they wanted, but most completed them just before the exam. The homework problems were divided based on similarity to the training problems, and assistance scores were calculated.
** ''[[Accelerated future learning]]'': The training was on electrical fields, and it was followed in the course by a unit on magnetic fields. Log data from the magnetic field homework was analyzed as a measure of acceleration of future learning.

=== Results ===
* [[Normal post-test]]
** ''Near [[transfer]], immediate'': The self-explanation condition demonstrating lower normalized assistance scores than the paraphrase condition, ''F''(1, 73) = 6.19, ''p'' = .02, ηp2 =.08. <center>[[Image:NTI_results.JPG]]</center>

* [[Robust learning]]
** ''Near transfer, retention'': Results on the measure were mixed. While there were no reliable main effects or interactions, the complete self-explanation group was marginally higher than the complete paraphrase condition (LSD, p = .06). Moreover, we analyzed the students’ performance on a homework problem that was isomorphic to the chapter exam in that they shared an identical deep structure (i.e., both analyzed the motion of a charged particle moving in two dimensions). The self-explanation had reliably lower normalized assistance scores than the paraphrase condition, ''F''(1, 27) = 4.07, ''p'' = .05, ηp2 = .13. <center>[[Image:FTR_results.JPG]]</center>
** ''Near and far transfer'':
** ''[[Accelerated future learning]]'': There was no effect for [[example]] completeness; however, the self-explanation condition demonstrating lower normalized assistance scores than the paraphrase condition, ''F''(1, 46) = 5.22, ''p'' = .03, ηp2 = .10. <center>[[Image:AFL_results.JPG]]</center>

=== Explanation ===
This study is part of the [[Interactive_Communication|Interactive Communication cluster]], and its hypothesis is a specialization of the IC cluster’s central hypothesis. The IC cluster’s hypothesis is that robust learning occurs when two conditions are met:

* The learning event space should have paths that are mostly learning-by-doing along with alternative paths where a second agent does most of the work. In this study, self-explanation comprises the learning-by-doing path and instructional explanations are ones where another agent (the author of the text) has done most of the work.

* The student takes the learning-by-doing path unless it becomes too difficult. This study tried (successfully, it appears) to control the student’s path choice. It showed that when students take the learning-by-doing path, they learned more than when they took the alternative path.

The IC cluster’s hypothesis actually predicts an attribute-treatment interaction (ATI) here. If some students were under-prepared and thus would find the self-explanation path too difficult, then those students would learn more on the instructional-explanation path. ATI analyzes have not yet been completed.

=== Further Information ===
==== Annotated bibliography ====
* Accepted as a poster at CogSci 2008
* Presentation to the Higher-order Cognition Group, led by Timothy J. Nokes, on March 3, 2008
* Presentation to the PSLC Advisory Board, December, 2006 [http://www.learnlab.org/uploads/mypslc/talks/hausmannabvisit2006bv2.ppt]
* Presentation to the NSF Follow-up Site Visitors, September, 2006
* Preliminary results were presented to the Intelligent Tutoring in Serious Games workshop, Aug. 2006 [http://projects.ict.usc.edu/itgs/talks/Hausmann_Generative%20Dialogue%20Patterns.ppt]
* Presentation to the NSF Site Visitors, June, 2006
* Poster presented at the annual meeting of the Science of Learning Centers, Oct. 2006.
* Symposium accepted to EARLI 2007
* Symposium accepted at AERA 2007
* Full-paper accepted at AIED 2007 [http://www.learnlab.org/uploads/mypslc/publications/hausmannvanlehn2007_final.pdf]
* Accepted as a poster at CogSci 2007
* Presentation to the NSF Site Visitors, May 31, 2007

==== References ====
# Anzai, Y., & Simon, H. A. (1979). The theory of learning by doing. ''Psychological Review, 86''(2), 124-140.
# Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. ''Cognitive Science, 13'', 145-182. [http://rt4rf9qn2y.scholar.serialssolutions.com/?sid=google&auinit=MTH&aulast=CHI&atitle=Self-explanations:+how+students+study+and+use+examples+in+learning+to+solve+problems&title=Cognitive+science&volume=13&issue=2&date=1989&spage=145&issn=0364-0213]
# Hausmann, R. G. M., & Chi, M. T. H. (2002). Can a computer interface support self-explaining? ''Cognitive Technology, 7''(1), 4-14. [http://www.pitt.edu/~bobhaus/hausmann2002.pdf]
# Lovett, M. C. (1992). Learning by problem solving versus by examples: The benefits of generating and receiving information. ''Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society'' (pp. 956-961). Hillsdale, NJ: Erlbaum.
# Schworm, S., & Renkl, A. (2002). Learning by solved example problems: Instructional explanations reduce self-explanation activity. In W. D. Gray & C. D. Schunn (Eds.), ''Proceedings of the 24th Annual Conference of the Cognitive Science Society'' (pp. 816-821). Mahwah, NJ: Erlbaum.[http://www.cognitivesciencesociety.org/confproc/gmu02/final_ind_files/schworm_renkl.pdf]
# Schworm, S., & Renkl, A. (2006). Computer-supported example-based learning: When instructional explanations reduce self-explanations. ''Computers & Education, 46,'' 426-445.

==== Connections ====
This project shares features with the following research projects:

* [[Booth | Knowledge component construction vs. recall (Booth, Siegler, Koedinger & Rittle-Johnson)]]
* [[Hausmann Study2 | The Effects of Interaction on Robust Learning (Hausmann & VanLehn)]]
* [[Craig observing | Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi)]]

==== Future plans ====
Our future plans for June 2007 - August 2007:
* Code transcripts for explanations and paraphrases.
* Link codes from transcript to knowledge components.
* Link transcript to steps in Andes log files.
* Submit to the International Journal of Artificial Intelligence in Education

File:ISE.JPG

2008-10-10T14:30:18Z

Bobhaus:

File:IPP.JPG

2008-10-10T14:30:09Z

Bobhaus: