Difference between revisions of "Assistance Dilemma English Articles"

From LearnLab
Jump to: navigation, search
(Study 1: Self-Explanation Study)
 
(13 intermediate revisions by 2 users not shown)
Line 14: Line 14:
 
|-
 
|-
 
! Start date study 2
 
! Start date study 2
| January 2008
+
| January 2009
 
|-
 
|-
 
! End date study 2
 
! End date study 2
| May 2008
+
| May 2009
 
|-
 
|-
 
! Learnlab
 
! Learnlab
Line 27: Line 27:
 
  Ruth Wylie, Teruko Mitamura, Ken Koedinger, and Jim Rankin
 
  Ruth Wylie, Teruko Mitamura, Ken Koedinger, and Jim Rankin
  
=== Abstract ===  
+
=== Background ===  
  
This project focuses on integrated theory development with respect to the Assistance Dilemma (Koedinger and Aleven, 2007). A key goal is to have a theory that is sufficient to make a priori predictions about what instructional treatments yield the most robust learning.
+
This project focuses on integrated theory development with respect to the Assistance Dilemma (Koedinger and Aleven, 2007), specifically the instructional principle of self-explanation. We explore whether having students self-explain and thereby adding difficulty aids or harms eventual learning outcomes.
  
The project focuses on exploring general principles of learning along two different dimensions of assistance: 1) task type: a) edit vs. b) selection; and 2) self-explanation: a) problems only vs. b) problems with prompts to self-explain.  We instantiate these principles within the domain of the English article system and within the English as a Second Language LearnLab. In this domain, standard practice often uses the (a) values on the dimensions above (edit of multiple knowledge components with no self-explanation) whereas emerging PSLC theory and analysis predict that the (b) values on the dimensions (selection on a subset of knowledge components with self-explanation) will lead to greater robust learning outcomes. These dimensions are interesting for exploring a priori predictions as to whether adding difficulties (removing assistance) during instruction aids or harms eventual learning outcomes.  For the task type dimension, our standing theoretical prediction is that the added demand of editing ahurt learning, while the added demand of self-explanation (dimension 2b) helps learning.
+
This work sets out to test, for the first time, whether the success of self-explanation extend to second language grammar learning, particularly the English article system (teaching students the difference between when to say “a dog” vs “the dog”). Previous work has demonstrated the effectiveness of self-explanation to enhance robust learning across multiple math and science domains and a variety of learners. Our goal is to better understand how, why, and when self-explanation works by investigating the effects of self-explanation on second language acquisition.
 +
 
 +
=== Self-Explanation Study 1 ===
  
  
=== Study 1: Self-Explanation Study 1 ===
 
==== Research Questions ====
 
 
==== Hypothesis ====
 
==== Hypothesis ====
 
Since the added process of self-explanation is generative, students in the self-explanation with menu condition will show greater learning gains on both normal and robust learning measures than those in the practice-only condition and free-form self-explanation condition. We hypothesize that the added difficulty of generating explanations in the free-form condition is extraneous and thus will result in less learning than the menu-based self-explanation conditions.
 
Since the added process of self-explanation is generative, students in the self-explanation with menu condition will show greater learning gains on both normal and robust learning measures than those in the practice-only condition and free-form self-explanation condition. We hypothesize that the added difficulty of generating explanations in the free-form condition is extraneous and thus will result in less learning than the menu-based self-explanation conditions.
Line 57: Line 57:
  
 
==== Results ====
 
==== Results ====
Students in all conditions showed significant pre to posttest improvement; however, there was no significant differences between conditions.
+
Students (n=63) in all conditions showed significant pre to posttest improvement.  While there were no significant main effects of condition, there was a significant aptitude-treatment interaction (F(2, 60) = 3.54, p = 0.036) of theoretical interest.  The dependent variable in this analysis was a learning efficiency score, which is a measure that combines learning gains and time-on-task into a single construct, and the article pre-test was the aptitude measure.  As shown in the figure below, lower aptitude students benefited more (i.e., learned more in less time) from the self-explanation manipulations whereas higher aptitude students benefited more from the practice-only condition.  This interaction is consistent with the theoretical notion that [[self-explanation]] is beneficial primarily during early construction and [[refinement]] of [[knowledge components]] whereas practice alone is better during knowledge [[strength|strengthening]] and [[fluency]] development.  It is also consistent with research on the [[worked example principle]] and particularly the "expertise reversal effect" (Kalyuga, Chandler, Tuovinen, & Sweller, 2001).
 +
 
 +
 
 +
 
 +
[[Image:Efficiency.jpg]]<br>
 +
 
 +
=== Self Explanation Study 2 ===
 +
The second study, conducted in Spring 2009 in Levels 3, 4, and 5 of the English LearnLab compared a practice-only tutoring system to a worked examples with self-explanation system. The goal of this study was to further investigate the effects of self-explanation on second language grammar learning. In the self-explanation conditions in the previous study, students were required to select an article <i>and</i> select an explanation. In this study, we isolated the two tasks such that students were either selecting articles (practice-only) <i>or</i> selecting an exlanation (worked examples with self-explanation).
 +
 
 +
====Independent Variables====
 +
 
 +
[[Image:Practice_only.jpg]]<br>
 +
Practice-Only Tutor:  Working with the practice-only tutor, students select an article (a, an, the, or no article) from a drop-down. Students receive immediate feedback on their selections and can request hints to help choose the right answer.
  
=== Study 2: Spacing of Knowledge Component Learning Events ===
+
[[Image:Worked_ex_with_SE.jpg]]<br>
The second study, conducted in Spring 2009 in Levels 3, 4, and 5 of the English LearnLab will investigate the effects of various temporal spacing between knowledge components. Students will be randomly assigned to one of four conditions:
+
Worked Example with Self-Explanation Tutor:In the worked example plus self-explanation tutor, students are given a completed sentence with the target article highlighted and are asked to select the feature of the sentence that best explains the article choice.
[[Image:Study 2-se.jpg]]
 
  
==== Hypothesis ====
+
====Dependent Variables====
Our hypothesis is that the current system presents knowledge components with too wide of spacing, thus narrowing the spacing (or moving to the right on the curve below) will lead to more robust learning efficiency.
+
Similar to Study 1, students were given a pretest and immediate posttest with items similar to those on which they were tutored. The pre and posttests contained both article selection and explanation selection items.
[[Image:Hypothesis-graph-timing.jpg|500px]]
 
  
=== Learning Measures ===
+
====Results====
Both studies will incorporate normal and robust learning measures. Examples of normal learning include: immediate pre and post-test learning gains on tutor-like tasks (e.g. editing tests for students using the edit interface; selection tests for students using the selection interface). Transfer tasks include pre/posttest learning gains on non-tutor tasks (e.g. editing tests for students using the selection tutor; selection tests for students using the edit tutor) as well as leveraging existing data collection efforts by the English LearnLab. For example, error analyses of student writing and recorded speech activities (RSAs) could be done to see how student article use changes. Furthermore, additional writing samples and RSAs can be analyzed in order to address questions surrounding long-term retention.
+
Results showed that students (n=101) in both conditions showed significant learning gains (F(1, 99) = 40.28, p < 0.001) but there was no difference between conditions.
  
In addition to the measures described above, based on feedback from the project presentation, we also plan to include a battery of behavioral surveys to measure how important students view correct grammar and article usage and general motivation.
+
[[Image:Learning_gain.jpg]]
  
 +
=== Contributions ===
 +
These studies are the first to experimentally test the effects of self-explanation in second language grammar learning, and the current results suggest that there are limitations to the benefits of self-explanation. This work highlights the need to continue investigating the self-explanation effect in new and different domains. It suggests there may be limitations to its applicability. Additionally, it is important to understand the source of difficulty within a domain and identify how self-explanations may or may not address it.  More research is needed to further specify those boundary conditions and relate them to basic understanding of cognitive processes.
  
 
=== References ===
 
=== References ===

Latest revision as of 19:08, 17 November 2009

PI Ruth Wylie, Teruko Mitamura, Ken Koedinger
Others with > 160 hours Jim Rankin
Start date study 1 September 2008
End date study 1 December 2008
Start date study 2 January 2009
End date study 2 May 2009
Learnlab English


Intelligent Writing Tutor

Ruth Wylie, Teruko Mitamura, Ken Koedinger, and Jim Rankin

Background

This project focuses on integrated theory development with respect to the Assistance Dilemma (Koedinger and Aleven, 2007), specifically the instructional principle of self-explanation. We explore whether having students self-explain and thereby adding difficulty aids or harms eventual learning outcomes.

This work sets out to test, for the first time, whether the success of self-explanation extend to second language grammar learning, particularly the English article system (teaching students the difference between when to say “a dog” vs “the dog”). Previous work has demonstrated the effectiveness of self-explanation to enhance robust learning across multiple math and science domains and a variety of learners. Our goal is to better understand how, why, and when self-explanation works by investigating the effects of self-explanation on second language acquisition.

Self-Explanation Study 1

Hypothesis

Since the added process of self-explanation is generative, students in the self-explanation with menu condition will show greater learning gains on both normal and robust learning measures than those in the practice-only condition and free-form self-explanation condition. We hypothesize that the added difficulty of generating explanations in the free-form condition is extraneous and thus will result in less learning than the menu-based self-explanation conditions.

Alternatively, generating self-explanations may require too much time and thus it may be better to provide students with extra examples and to implicitly learn the rules through the practice-only condition.

In addition, while self-explanation has proved to lead to greater learning in other domains, this would be the one of the first, if not the first, study to empirically examine its effects in second language grammar acquisition.

Independent Variables

Picture 1.jpg
Image 1: Practice-Only Tutor - Using this tutor, students select an article (a, an, the, no article) from each sentence to complete the sentence.

Picture 7.jpg
Image 2: Menu-based Self-Explanation Tutor - Using this tutor, students select an article (a, an, the, no article) from each sentence and choose the feature of the sentence that best explains the article choice.

Picture 2.jpg
Image 3: Free-form Self-Explanation Tutor- Using this tutor, students select an article (a, an, the, no article) from each sentence and write an explanation for why they made their choice.

Dependent Variables

All students completed a computer-based pre and posttest that consisted of article-only and article with explanation items. In the article-only items, students chose an article from a dropdown menu to complete the sentence. In the article with explanation items, students chose an article to complete the sentence and then chose the feature or rule that explained their answer. No hints were available during the tests, and students did not receive feedback on their answers.

Results

Students (n=63) in all conditions showed significant pre to posttest improvement. While there were no significant main effects of condition, there was a significant aptitude-treatment interaction (F(2, 60) = 3.54, p = 0.036) of theoretical interest. The dependent variable in this analysis was a learning efficiency score, which is a measure that combines learning gains and time-on-task into a single construct, and the article pre-test was the aptitude measure. As shown in the figure below, lower aptitude students benefited more (i.e., learned more in less time) from the self-explanation manipulations whereas higher aptitude students benefited more from the practice-only condition. This interaction is consistent with the theoretical notion that self-explanation is beneficial primarily during early construction and refinement of knowledge components whereas practice alone is better during knowledge strengthening and fluency development. It is also consistent with research on the worked example principle and particularly the "expertise reversal effect" (Kalyuga, Chandler, Tuovinen, & Sweller, 2001).


Efficiency.jpg

Self Explanation Study 2

The second study, conducted in Spring 2009 in Levels 3, 4, and 5 of the English LearnLab compared a practice-only tutoring system to a worked examples with self-explanation system. The goal of this study was to further investigate the effects of self-explanation on second language grammar learning. In the self-explanation conditions in the previous study, students were required to select an article and select an explanation. In this study, we isolated the two tasks such that students were either selecting articles (practice-only) or selecting an exlanation (worked examples with self-explanation).

Independent Variables

Practice only.jpg
Practice-Only Tutor: Working with the practice-only tutor, students select an article (a, an, the, or no article) from a drop-down. Students receive immediate feedback on their selections and can request hints to help choose the right answer.

Worked ex with SE.jpg
Worked Example with Self-Explanation Tutor:In the worked example plus self-explanation tutor, students are given a completed sentence with the target article highlighted and are asked to select the feature of the sentence that best explains the article choice.

Dependent Variables

Similar to Study 1, students were given a pretest and immediate posttest with items similar to those on which they were tutored. The pre and posttests contained both article selection and explanation selection items.

Results

Results showed that students (n=101) in both conditions showed significant learning gains (F(1, 99) = 40.28, p < 0.001) but there was no difference between conditions.

Learning gain.jpg

Contributions

These studies are the first to experimentally test the effects of self-explanation in second language grammar learning, and the current results suggest that there are limitations to the benefits of self-explanation. This work highlights the need to continue investigating the self-explanation effect in new and different domains. It suggests there may be limitations to its applicability. Additionally, it is important to understand the source of difficulty within a domain and identify how self-explanations may or may not address it. More research is needed to further specify those boundary conditions and relate them to basic understanding of cognitive processes.

References

1. Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp.185-205). Cambridge, MA: MIT Press.
2. Celce-Murcia, Marianne, and Larsen-Freeman, Diane. (1983). The Grammar Book: An ESL/EFL Teacher’s Course. Rowley, Massachusetts: Newbury House Publishers.
3. Elson, Allegra B. Fossilized language forms: Implications and a search for solutions in an adult English as second language classroom. A PALPIN Inquiry Project, 2004.
4. Ericsson, K. A, & Simon, H. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press
5. Hawkins, J. A. (1991). On (in) definite articles: implicatures and (un) grammaticality prediction. Journal of Linguistics(27), 405-442.
6. Knowledge Component Hypothesis (2007, April 30). In PSLC Theory Wiki. Retrieved March 27, 2008, from http://learnlab.org/research/wiki/index.php/Knowledge_component_hypothesis
7. Koedinger, K., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review. 19(3) 239-264.
8. Level 4 Writing Objectives. (2007, Novermber 13). Retrieved March 19, 2008, from http://learnlab.org/learnlabs/english/Level_4_Objectives/Objectives_Wr4fin.dco
9. Liu, D. & Gleason, J. (2002). Acquisition of the Article THE by Nonnative Speakers of English, An Analysis of Four Nongeneric Uses. SSLA, 24, 1-26.
10. Master, P. (1997). The English Article System: Acquisition, Function, and Pedagogy. System. 25,(2) 215-232.
11. Pavlik Jr., P.I., and Anderson, J. R. (2005). Practice and forgetting effects on vocabulary memory: An activation-based model of the spacing effect. Cognitive Science, 29(4), 559-586.
12. Schooler, JW & Engstler-Schooler, TY. (1990). Verbal overshadowing of visual memories: some things are better left unsaid. Cognitive Psychology. 22(1):36-71.
13. VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W.B. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 21(3), 209-249.
14. Wylie, R. (2007) Are we asking the right questions? Understanding which tasks lead to the robust learning of English grammar. Accepted as a Young Researchers Track paper at the 13th International Conference on Artificial Intelligence in Education. Marina del Rey, California. July 9 – 13, 2007.
15. Wylie, R. (2008) Putting a/the stake in the ground: Making a priori predictions of student learning. Accepted as a Young Researchers Track paper at Intelligent Tutoring Systems 2008. Montreal, Canada. June 23 – 27, 2008.