Difference between revisions of "REAP Study on Personalization of Readings by Topic (Fall 2006)"

From LearnLab
Jump to: navigation, search
(Further Information)
Line 122: Line 122:
[[Stoichiometry_Study | Studying the Learning Effect of Personalization and Worked Examples in the Solving of Stoichiometry Problems (McLaren, Koedinger & Yaron)]]
[[Stoichiometry_Study | Studying the Learning Effect of Personalization and Worked Examples in the Solving of Stoichiometry Problems (McLaren, Koedinger & Yaron)]]
Plans for June 2007 - December 2007:
*Analyze [[Transfer]], [[Long-term retention]] test results.
*Write full conference or journal paper describing findings.
=== Annotated bibliography ===
=== Annotated bibliography ===

Revision as of 18:28, 10 January 2008

REAP Study on Personalization of Readings for Increased Interest

Logistical Information

Contributors Maxine Eskenazi, Alan Juffs, Michael Heilman, Kevyn Collins-Thompson, Lois Wilson, Jamie Callan
Study Start Date September 11, 2006
Study End Date November 21, 2006
Learnlab Courses English Language Institute Reading 4 (ESL LearnLab)
Number of Students 35
Total Participant Hours (est.) 270
Data in Datashop no


This paper discusses the enhancement of the REAP tutor to allow for personalization of reading materials by topic in order to increase interest and motivation. In this work, the term “personalization” refers to the selection of practice readings in order to match a student’s interests.

During each training session with REAP, students work through a series of readings, each of which is followed by practice exercises for the target words in the reading. While reading a passage, students are able to access dictionary definitions for any word in a reading either by clicking on a highlighted target word or by typing a word into a box in the lower-left corner of the screen. The target words in the readings are also highlighted because highlighting may increase the use of dictionary definitions, thus encouraging students to coordinate multiple sources of information about a word’s meaning—namely, the implicit context around words and the explicit definitions of words.

A problem discovered in past studies with REAP is that many students spend only a brief amount of time on a reading and do not deeply process the text. Students often only read the dictionary definition for target words rather than attempting to process the entire context around the words. Inferring the meaning of vocabulary from context is a seemingly important strategy that is not used by such students. This behavior is likely due to a desire to perform well on post-reading practice exercises and post-test, which can be viewed as forms of extrinsic motivation. Intrinsically motivated students who are more interested in a reading are more likely to read the entire text and to use context to learn the meaning of unknown vocabulary. Therefore, personalization that increases intrinsic motivation could lead to deeper processing of context and better learning of vocabulary.

Passive Active Interactive
Explicit (general) Dictionary Definitions Practice Exercises
Implicit (instance) Interpreting meaning in context while reading Sentence Production (assessment) Practice Exercises


Intrinsic Motivation: Motivation to learn for learning's own sake rather than some external goal.

Extrinsic Motivation: Motivation for learn in order to satisfy an external goal, such as completing a task or passing an assessment.

Research question

Do the benefits of personalization of practice readings by topics of interest outweigh the costs in a tutoring system for ESL vocabulary practice?

Dependent variables

Normal post-test scores

Normal post-test scores for practiced words only

Long-term retention test scores, same post-test but administered months later.

Evidence of Transfer: sentence production tasks for target words, correct use of words in writing assignments for other courses.

Independent variables

Personalization of readings by topics of interest. In the control condition, the tutor did not use potential personal interest as a factor in its selection of reading materials. In the treatment condition, the tutor did use interest as a factor. All other selection criteria were the same in both conditions. Time on task was also the same.


Since intrinsic motivation seems to be important in language learning, the benefits of personalization will outweigh the costs.


Students in the treatment condition with personalization performed better on average (M=35.5%, SD=14.9%) in terms of overall post-test scores compared to students in the control condition (M=27.1%, SD=17.2%). Further analysis, including statistical tests for significance, is forthcoming....



There is evidence that the difference in post-test scores is due to increased interest leading to deeper processing of the reading practice texts.

Responses to questionnaires following each reading show the interest level of students using the REAP tutor. The questionnaires asked students to indicate on a scale from one to five their interest inr the preceding text. The distributions of post-reading interest ratings for students in the treatment and control conditions are shown in Figures 1 and 2.

File:Interest combined.PNG

Students were also given an exit survey during their last week of practice with the tutor that asked them, among other questions, for to indicate whether they agreed with the statement, “Most of the readings were interesting.” The ratings were on a scale from one to five, with five indicating strong agreement and one indicating strong disagreement. Exit survey interest ratings by students in the treatment condition were significantly higher (p<0.05) than the ratings by students in the control condition. The mean response for students who received personalized readings was 3.18, while it was 2.65 for students in the control condition.

Further analysis of post-test scores reveals that students did learn more of the words that they actually practiced in REAP. The post-test contained 40 questions for target vocabulary words. Many of the students did not practice 40 words, so performance on practiced words alone was analyzed. Students in the treatment condition scored higher (N=16, M=50.3, SD=20.1) on questions for words seen in readings than did students in the control condition (N=19, M=32.4, SD=18.9). A two-tailed t-test for independent means verified that this result is statistically significant (t=2.719, df=33, p=0.005). The difference of scores between the two groups was 17.9% (95% CI = 4.5%, 31.3%), which corresponds to a large effect size of 0.85. This result indicates that personalization improved learning for the words that students saw in readings, which is in line with previous findings that intrinsic motivation leads to improved learning.

File:Post just practice.PNG

However, students in the treatment condition that included personalization saw fewer words in their training sessions (N=16, M=12.0 , SD=1.13) than students in the control condition (N=19, M=16.3, SD=0.87) (t=-2.9, df=33, p=0.006). Average time on task was essentially the same for students in both conditions. Students in the treatment condition spent slightly longer on each reading. The main reason, however, for the difference in the average total number of words practiced was that students for whom the tutor provided personalized instruction saw fewer words (M=3.41, SD=0.55) per practice reading passage than students in the control condition (M=4.07, SD=0.83) (t=2.929, df=33, p=0.006).

Thus, when the tutor used personalization as a factor in the selection of readings, it chose readings that were less valuable according to other factors. Specifically, this result shows that by personalizing instruction, the tutor was not able to provide practice for as many words. Of course, the practice that it did provide was better, as is shown in the previous result that for words student did practice, personalization appeared to increase learning.

The reduced number of target words per text with personalization is a technical issue which can be avoided in a straightforward manner by increasing the size of the database of readings. With more readings, the tutor can find texts that both have ample target words and cover topics of personal interest.

File:Words per reading.PNG

There is a possibility that the students in the treatment condition who were seeing fewer words in each reading were learning more of the words simply because they had fewer to learn per reading. To rule out this hypothesis, regression analyses (multiple linear regression) with overall post-test performance and performance for practiced words as the dependent variables. In both regression analyses, the number of target words per reading was not a significant predictor of performance. In fact, the number of target words per document was slightly positively correlated with post-test performance in both cases. This result seems to rule out the possibility that students were learning more target words in the treatment condition because they were seeing fewer words.

Long-term retention test results showed no reliable differences because of a small sample size. The test was administered to students who stayed in the ELI in the subsequent semester, which constituted only a fraction of the original sample.

Further Information

The following study addresses a different form of personalization, by which interactions with the learner (e.g., instructions, directions) are conducted using casual and direct rather than formal language:

Studying the Learning Effect of Personalization and Worked Examples in the Solving of Stoichiometry Problems (McLaren, Koedinger & Yaron)

Annotated bibliography

Heilman, M., Juffs, A., & Eskenazi, M. (To Appear). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)

Clark, R. C. and Mayer, R. E. (2003). e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer.

Cordova, D. I. & Lepper, M. R. (1996). Intrinsic Motivation and the Process of Learning: Beneficial Effects of Contextualization, Personalization, and Choice. Journal of Educational Psychology. Vol. 88,l No. 4, 715-730.

Lepper, M. (1988). Motivational Considerations in the Study of Instruction. Cognition and Instruction. 5(4), 289-309.

Heilman, M., Juffs, A., & Eskenazi, M. (To Appear). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)