Assistance Dilemma English Articles

PI	Ruth Wylie, Teruko Mitamura, Ken Koedinger
Others with > 160 hours	Jim Rankin
Start date study 1	September 2008
End date study 1	December 2008
Start date study 2	January 2008
End date study 2	May 2008
Learnlab	English

Intelligent Writing Tutor

Ruth Wylie, Teruko Mitamura, Ken Koedinger, and Jim Rankin

Abstract

This project focuses on integrated theory development with respect to the Assistance Dilemma (Koedinger and Aleven, 2007). A key goal is to have a theory that is sufficient to make a priori predictions about what instructional treatments yield the most robust learning.

The project focuses on exploring general principles of learning along two different dimensions of assistance: 1) task type: a) edit vs. b) selection; and 2) self-explanation: a) problems only vs. b) problems with prompts to self-explain. We instantiate these principles within the domain of the English article system and within the English as a Second Language LearnLab. In this domain, standard practice often uses the (a) values on the dimensions above (edit of multiple knowledge components with no self-explanation) whereas emerging PSLC theory and analysis predict that the (b) values on the dimensions (selection on a subset of knowledge components with self-explanation) will lead to greater robust learning outcomes. These dimensions are interesting for exploring a priori predictions as to whether adding difficulties (removing assistance) during instruction aids or harms eventual learning outcomes. For the task type dimension, our standing theoretical prediction is that the added demand of editing ahurt learning, while the added demand of self-explanation (dimension 2b) helps learning.

Study 1: Self-Explanation Study 1

Hypothesis

Since the added process of self-explanation is generative, students in the self-explanation with menu condition will show greater learning gains on both normal and robust learning measures than those in the practice-only condition and free-form self-explanation condition. We hypothesize that the added difficulty of generating explanations in the free-form condition is extraneous and thus will result in less learning than the menu-based self-explanation conditions.

Alternatively, generating self-explanations may require too much time and thus it may be better to provide students with extra examples and to implicitly learn the rules through the practice-only condition.

In addition, while self-explanation has proved to lead to greater learning in other domains, this would be the one of the first, if not the first, study to empirically examine its effects in second language grammar acquisition.

Independent Variables

Image 1: Practice-Only Tutor - Using this tutor, students select an article (a, an, the, no article) from each sentence to complete the sentence.

Image 2: Menu-based Self-Explanation Tutor - Using this tutor, students select an article (a, an, the, no article) from each sentence and choose the feature of the sentence that best explains the article choice.

Image 3: Free-form Self-Explanation Tutor- Using this tutor, students select an article (a, an, the, no article) from each sentence and write an explanation for why they made their choice.

Dependent Variables

All students completed a computer-based pre and posttest that consisted of article-only and article with explanation items. In the article-only items, students chose an article from a dropdown menu to complete the sentence. In the article with explanation items, students chose an article to complete the sentence and then chose the feature or rule that explained their answer. No hints were available during the tests, and students did not receive feedback on their answers.

Results

Students in all conditions showed significant pre to posttest improvement; however, there was no significant differences between conditions. Analysis using efficiency scores (a measure that combines learning gains and time-on-task into a single construct) revealed a significant aptitude by treatment interaction (F(2, 60) = 3.54, p = 0.036)

Study 2: Self Explanation Study 2

The second study, conducted in Spring 2009 in Levels 3, 4, and 5 of the English LearnLab compared a practice-only tutoring system to a worked examples with self-explanation system.

Hypothesis

Our hypothesis is that the current system presents knowledge components with too wide of spacing, thus narrowing the spacing (or moving to the right on the curve below) will lead to more robust learning efficiency.

Learning Measures

Both studies will incorporate normal and robust learning measures. Examples of normal learning include: immediate pre and post-test learning gains on tutor-like tasks (e.g. editing tests for students using the edit interface; selection tests for students using the selection interface). Transfer tasks include pre/posttest learning gains on non-tutor tasks (e.g. editing tests for students using the selection tutor; selection tests for students using the edit tutor) as well as leveraging existing data collection efforts by the English LearnLab. For example, error analyses of student writing and recorded speech activities (RSAs) could be done to see how student article use changes. Furthermore, additional writing samples and RSAs can be analyzed in order to address questions surrounding long-term retention.

In addition to the measures described above, based on feedback from the project presentation, we also plan to include a battery of behavioral surveys to measure how important students view correct grammar and article usage and general motivation.

References

1. Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe and A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp.185-205). Cambridge, MA: MIT Press.
2. Celce-Murcia, Marianne, and Larsen-Freeman, Diane. (1983). The Grammar Book: An ESL/EFL Teacher’s Course. Rowley, Massachusetts: Newbury House Publishers.
3. Elson, Allegra B. Fossilized language forms: Implications and a search for solutions in an adult English as second language classroom. A PALPIN Inquiry Project, 2004.
4. Ericsson, K. A, & Simon, H. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press
5. Hawkins, J. A. (1991). On (in) definite articles: implicatures and (un) grammaticality prediction. Journal of Linguistics(27), 405-442.
6. Knowledge Component Hypothesis (2007, April 30). In PSLC Theory Wiki. Retrieved March 27, 2008, from http://learnlab.org/research/wiki/index.php/Knowledge_component_hypothesis
7. Koedinger, K., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review. 19(3) 239-264.
8. Level 4 Writing Objectives. (2007, Novermber 13). Retrieved March 19, 2008, from http://learnlab.org/learnlabs/english/Level_4_Objectives/Objectives_Wr4fin.dco
9. Liu, D. & Gleason, J. (2002). Acquisition of the Article THE by Nonnative Speakers of English, An Analysis of Four Nongeneric Uses. SSLA, 24, 1-26.
10. Master, P. (1997). The English Article System: Acquisition, Function, and Pedagogy. System. 25,(2) 215-232.
11. Pavlik Jr., P.I., and Anderson, J. R. (2005). Practice and forgetting effects on vocabulary memory: An activation-based model of the spacing effect. Cognitive Science, 29(4), 559-586.
12. Schooler, JW & Engstler-Schooler, TY. (1990). Verbal overshadowing of visual memories: some things are better left unsaid. Cognitive Psychology. 22(1):36-71.
13. VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W.B. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 21(3), 209-249.
14. Wylie, R. (2007) Are we asking the right questions? Understanding which tasks lead to the robust learning of English grammar. Accepted as a Young Researchers Track paper at the 13th International Conference on Artificial Intelligence in Education. Marina del Rey, California. July 9 – 13, 2007.
15. Wylie, R. (2008) Putting a/the stake in the ground: Making a priori predictions of student learning. Accepted as a Young Researchers Track paper at Intelligent Tutoring Systems 2008. Montreal, Canada. June 23 – 27, 2008.