REAP Study on Personalization of Readings by Topic (Fall 2006)
REAP Study on Personalization of Readings for Increased Interest
Intrinsic motivation and personal interest have been shown to lead to deeper processing, the use of learning strategies, and better learning outcomes (Lepper, 1988). Therefore, it seems reasonable to try to increase interest in a tutoring environment. Previous studies have shown both positive (Cordova & Lepper, 1997) and negative (Clark & Mayer, 2003) effects of attempts to increase interest. Clark and Mayer (2003) state that adding interest but irrelevant material may distract or disrupt the learner.
This study investigated the effects of personalizing practice readings by topic. The REAP tutor for ESL vocabulary can prefer practice readings that match personal interests as specified in a questionnaire. However, if this preference is included as a factor in reading selection, then other factors, such as the density of practice opportunities in a reading, are necessarily given less weight. Finding a reading for a student is like searching for an optimal point in a multidimensional space. In practice, the tutor never finds optimal readings, but must weigh different factors against each other (e.g., reading difficulty, length, density of practice opportunities, etc.). Adding personalization as another factor may significantly affect the tutor's ability to find readings of high quality according to other factors.
Intrinsic Motivation: Motivation to learn for learning's own sake rather than some external goal.
Extrinsic Motivation: Motivation for learn in order to satisfy an external goal, such as completing a task or passing an assessment.
Personalization: The preference of practice materials to match the personal interests of the learner. In this context, this term is used as it is in information retrieval rather than as in Mayer's work (Clark & Mayer, 2003) where it means using casual, direct language.
Do the benefits of personalization of practice readings by topics of interest outweigh the costs in a tutoring system for ESL vocabulary practice?
Immediate post-test scores for practiced words.
Number of words practiced
Overall post-test scores (essentially a product of the previous two)
Long-term retention test scores similar to post-test but administered months later.
Transfer of knowledge: sentence production tasks for target words, correct use of words in writing assignments for other courses.
Personalization of readings by topics of interest. In the control condition, the tutor did not use potential personal interest as a factor in its selection of reading materials. In the treatment condition, the tutor did use interest as a factor. All other selection criteria were the same in both conditions. Time on task was also the same.
Since intrinsic motivation seems to be important in language learning, the benefits of personalization will outweigh the costs.
The interest level of students using the REAP tutor is evident in their response to a questionnaire following each reading that asked them to indicate on a scale from one to five their interest in the preceding text. The distributions of post-reading interest ratings for students in the treatment and control conditions are shown in Figures 1 and 2.
Figures 1&2: File:Interest combined.PNG
Students were also given an exit survey during their last week of practice with the tutor that asked them, among other questions, for to indicate whether they agreed with the statement, “Most of the readings were interesting.” The ratings were on a scale from one to five, with five indicating strong agreement and one indicating strong disagreement. Exit survey interest ratings by students in the treatment condition were significantly higher (p<0.05) than the ratings by students in the control condition. The mean response for students who received personalized readings was 3.18, while it was 2.65 for students in the control condition.
The effect of personalization on student learning was measured by post-test cloze questions for vocabulary words. Two measurements were made for each group. First, the percentage correct of post-test questions was measured for target words that had appeared in the readings for each student. Second, the overall post-test score was measured. Additionally, student progress through the curriculum was measured for each student by the total number of words from his or her individual target vocabulary list that had appeared in at least one reading. Students in the treatment condition scored higher (N=16, M=50.3, SD=20.1) on questions for words seen in readings than did students in the control condition (N=19, M=32.4, SD=18.9). A two-tailed t-test for independent means verified that this result is statistically significant (t=2.719, df=33, p=0.005). The difference of scores between the two groups was 17.9% (95% CI = 4.5%, 31.3%), which corresponds to a large effect size of 0.85. This result indicates that personalization improved learning for the words that students saw in readings, which is in line with previous findings that intrinsic motivation leads to improved learning.
However, students in the treatment condition that included personalization saw fewer words in their training sessions (N=16, M=12.0 , SD=1.13) than students in the control condition (N=19, M=16.3, SD=0.87) (t=-2.9, df=33, p=0.006). Average time on task was essentially the same for students in both conditions. Students in the treatment condition spent slightly longer on each reading. The main reason, however, for the difference in the average total number of words practiced was that students for whom the tutor provided personalized instruction saw fewer words (M=3.41, SD=0.55) per practice reading passage than students in the control condition (M=4.07, SD=0.83) (t=2.929, df=33, p=0.006). Thus, when the tutor used personalization as a factor in the selection of readings, it chose readings that were less valuable according to other factors. Specifically, this result shows that by personalizing instruction, the tutor was not able to provide practice for as many words. Of course, the practice that it did provide was better, as is shown in the previous result that for words student did practice, personalization appeared to increase learning.
Despite practicing fewer words, students in the treatment condition with personalization still performed better on average (M=35.5%, SD=14.9%) in terms of overall post-test scores compared to students in the control condition (M=27.1%, SD=17.2%). The improvement of average overall post-test scores in the treatment condition was only 8.4% (95% CI = -2.8%, 19.5%), which corresponds to a medium effect size of 0.51. However, this difference is not statistically significant (p=0.14). Therefore, the null hypothesis that personalization has no effect on overall post-test scores cannot be rejected.
Results pending acceptance of paper.
Submitted paper to appear here.
Clark, R. C. and Mayer, R. E. (2003). e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer.
Cordova, D. I. & Lepper, M. R. (1996). Intrinsic Motivation and the Process of Learning: Beneficial Effects of Contextualization, Personalization, and Choice. Journal of Educational Psychology. Vol. 88,l No. 4, 715-730.
Lepper, M. (1988). Motivational Considerations in the Study of Instruction. Cognition and Instruction. 5(4), 289-309.