Learning ESL Vocabulary with Context and Definitions: Order Effects and Self-Generation

From LearnLab
Jump to: navigation, search

Learning Vocabulary in an ESL Classroom: Definition Order Effects and Self Generation

Summary Table

Project title Learning ESL Vocabulary Using Context and Definitions: Order Effects and Self-Generation
Principal Investigator Michal Balass, M.S., Jessica R. Nelson, M.S. (Graduate Students, University of Pittsburgh
Co-PIs Dr. Charles A. Perfetti (faculty, University of Pittsburgh
Study start and end dates Study 1: September - December 2008
Study 2: January - April 2009
Learnlab ESL, Reading courses (levels 4, 5)
Number of participants 140
Total Participant Hours 140
Datashop Data files not yet available
Current status (May 2009) Data analysis of Study 1 and Study 2 is under progress


Previous research in our lab has shown that there was a benefit for vocabulary learning when learners were exposed to a vocabulary word in a sentence and asked to generate/guess a definition for the word based on the sentence before they were given the word's definition (Balass et al., in prep). This could be due to either a benefit from receiving the word in context before getting the word's definition (an order effect) or due to the deep generation task of constructing a definition for the word, or both. The experiment we are conducting in the classroom is designed to test the effects of both deep generation tasks and order-of-exposure of definitions and contexts using the REAP software tool. It is also designed to test whether these findings extend to a real classroom situation with ESL learners reading longer texts. All learners will get both a context and a definition for each word during learning; only the order of exposure and the generation task will vary. They will then be tested on the words immediately afterwards with multiple choice tests of spelling, best use of the word in a sentence, and related words, as well as a multiple choice comprehension question to ensure they read the passage and a brief definition generation question. They will also be tested again on each set of words the following week to test retention. We predict that the condition in which they receive context first and are required to do the deep generation task of generating a definition will result in the most robust learning.

Background and Significance

Many studies of vocabulary learning have demonstrated that dictionary definitions are beneficial for meaning acquisition when the definitions are presented with context passages or sentences (e.g., Bolger, Balass, Landen & Perfetti, 2008; Scott & Nagy, 1997, Nist & Olejnik, 1995) rather than in isolation (McKeown, 1993). As McKeown illustrated in her analysis of dictionary definitions, definitions tend to be abstract and disjoint pieces of meaning information that need to be supplemented with more concrete instances of word use for them to be effective for learning. For example, in a study with adult native English speakers, Bolger et al., showed greater learning gains when definitions were supplemented with single sentence contexts on a variety of word learning measures (e.g., spelling, sentence discrimination, and definition generation). There is also evidence to suggest that the order in which definitions and contexts are presented to the learner affects learning. Balass, Bolger, and Perfetti (in prep) illustrated greater vocabulary learning gains when learners were exposed to contexts followed by definitions than they did when exposed to definitions followed by contexts. In this study, learning was only measured by a definition generation task; learners were more accurate in their generated definition for a target word when the word was first encountered in a context followed by its definition. This finding suggests that learners benefited from seeing context early during learning because they could later integrate the more abstract definition with a concrete experience with the word. Balass et al., refer to this finding as a definition order effect.

As mentioned previously, learners in the study were required to generate a definition for the target word. Specifically, they generated a definition on each learning trial, where there were four trials per target word. A definition was presented on the first trial followed by three trials of context sentences or on the forth trial preceded by three trials of context sentences. Learners were asked to generate a definition for the word on each trial in order for the experimenters to be able to track their meaning learning progress. Given this manipulation, the results observed by Balass et al., (in prep) may not be due to an order effect, but rather a benefit from the generation process itself. Thus, another possible explanation for the results observed is that learners benefited from the generation process itself during learning, and more so when they had to generate a definition using only context, without the benefit of having seen a dictionary definition. In support of this possibility, previous learning studies have illustrated that learners show robust learning gains when asked to generate their own self-explanations for targeted material than when asked to just read the information (Chi, 1994; Roy & Chi, 1995).

These generation and self-explanation findings are especially critical for instruction of ESL learners because studies have shown that these learners benefit from active word-focused and high level of engagement tasks (e.g., sentence generation, definition generation) for learning vocabulary (Laufer, 2003; Nation, 2001; Schmitt, 2008). Moreover, not only do these tasks lead to more robust meaning learning and retention, they also enhance L2 reading development (Pichette, 2005). Thus, ESL learners may benefit from learning vocabulary using a similar paradigm used by Balass, et al., (in prep), where generating a definition from context for an unfamiliar word involves a high-level of engagement from the learner.

In addition, under some circumstances, withholding information to allow a student to work through a problem can be more effective than giving the necessary information right away, leading to "the assistance dilemma" in instruction: finding the optimal balance of giving and withholding information in a learning domain (Koedinger & Aleven, 2007). If self-generation is important in this learning situation, it is possible that the order of exposure for definitions and contexts does not matter as long as the learner engages with the material using an active generation process while further instructional materials are temporarily withheld. Seeing a definition first may be as beneficial as seeing a context first if the learner is required to actively generate a context sentence based on the definition before being given a context. Our proposed experimental manipulation will decouple the effect of order and the effect of information generation.

In order to investigate the order and generation effects, we will be using REAP (Collins-Thompson & Callan, 2005) to implement our study. REAP is a search engine that an instructor or an experimenter can use to find authentic context passages from an open-corpus (the Web). For example, one can use REAP to find contexts with specific constraints, i.e., a passage with a particular target word. This feature of REAP is specifically significant because it allows the experimenter to use authentic written material to examine meaning acquisition. This feature is an improvement of previous studies with adults (e.g., Balass et al., in prep; Bolger et al. 2008) that used highly controlled experimenter generated contexts that may not approximate how a learner acquires new vocabulary.

To summarize, our research goal is to address the following question: Does the generation of sentence contexts or definitions improve learning and long-term retention of vocabulary terms in ESL students, and does the order of exposure to definition and context play a role? In addressing these research questions we will be extending previous work with native speakers of English in order to suggest more optimal robust learning paths for ESL learners to acquire new word meaning.


Research Questions

The purpose of this research is to determine the most effective use of definitions and contexts in an automated vocabulary learning environment implemented in an ESL classroom. This knowledge will help us understand the cognitive processes underlying vocabulary learning. Specifically, the study will determine the effect of:

1. order (definition or context first)

2. self-generation (shallow vs. deep)


Study 1:


This study is an in vivo experiment. We are testing the outcome of vocabulary learning for words in the Academic Word List currently taught in the existing ESL curriculum at the University of Pittsburgh, using students in the level 4 and 5 classrooms. The benefit of using this population is that we can extend laboratory findings to a real classroom learning situation. Additionally, learning English vocabulary is important for L2 learners to gain English proficiency. A 2 (order: definition vs. passage first) x 2 (deep vs. shallow generation) experimental design has been incorporated into the existing REAP software resulting in four learning paths based on our experimental hypotheses:

(1) dictionary definition ==> sentence generation ==> context passage

(2) context passage ==> definition generation ==> dictionary definition

(3) dictionary definition ==> definition summary ==> context passage

(4) context passage ==> passage topic summary ==>dictionary definition

Conditions 1 and 2 reflect a deep generation process (generating a sentence from a definition or generating a definition from a context), whereas conditions 3 and 4 use a shallow generation task for control (summarizing the definition or the topic of the passage). Study1design.jpg

Generation Responses Coding:

The self-generation responses across conditions did not all have clear correct or incorrect answers, but rather had varying degrees of complexity. For example some of the generation tasks involved generating an original sentence, while other conditions asked for a definition generation. The sentence generation required a broader scale than just correct or incorrect as in the definition generation tasks. Therefore a rubric was created to provide an objective measure for rating the participants’ responses. The responses within each conditon were coded using the rubric. The rubric below details each measure and scale used for the different conditions.

Context/Deep (Generation of a definition based on the context passage)

(0) Incorrect definition
(1) Correct Definition
Context related
(0) Unrelated to Context
(1) Related to context

Context/Shallow (Generation of passage topic summary)

How related/detailed response is to passage.
(1) Response does not relate to any part of passage
(2) Response provides some detail to passage but may leave out important details or provides some inaccurate information
(3) Response accurately conveys main idea of passage
(1) Verbatim
(2) Some original words
(3) All original

Definition/Deep (Generation of a sentence with target word given the definition)

Word Use in sentence:
(1) Poor use of word,
(2) Awkward sounding/ not native like
(3) Good use of word
Word meaning:
(1) Sentence does not convey meaning of word
(2) Sentence conveys some meaning of word
(3) Sentence provides good context for word
Related to given definition
(0) Not related
(1) Related to definition

Definition/Shallow (Generation of a definition summary given the definition)

(0) Incorrect definition
(1) Correct definition
Original Words
(0) Repeated definition
(1) Generated some original words
(2) generated all original words

Study 2:

This study is an in vivo experiment. We are testing the outcome of vocabulary learning for low frequency words that are not currently taught in the existing ESL curriculum at the University of Pittsburgh. We are using students in the level 4 classrooms to examine the effects of definition order during new word learning. In study 1, we failed to replicate the results from previous studies showing a definition order effect. This may have been due to a complicated study design. Thus, to examine the effects of definition order more closely, a single variable definition position (first, second, third)experimental design had been incorporated into the existing REAP software resulting in three learning paths:

(1) dictionary definition ==> context sentence ==> context sentence

(2) context sentence ==> dictionary definition ==> context sentence

(3) context sentence ==> context sentence ==> dictionary definition



A. Study 1: Definition Order and Self-Generation Slide1.jpg

B. Study 2: Definition Order Slide2.jpg

Self Generation Data

The study 2 self generation responses ( a sentence using target word) were coded using a rubric. The sentences were coded for the participants’ word use and meaningfulness of the target word. A paired t-test determined that participants performed better on both measures of the self generation task when they were familiar with the word before the training, word use: t(57) = -4.86, p < .001; meaningfulness: t(57) = -4.55, p < .001. We chose to use only the unfamiliar coded data in further analysis to represent words learned in the experimental training conditions.


The measure word use significantly correlated with definition generation scores, r(57) = .431, p < .001, related word scores, r(57) = .26, p = .47, and sentences discrimination(57) = .401, p = .002. Meaningfulness correlated positively with definition generation, r(57) = .366, p = .004, related words, r(57) = .312, p = .016, and sentence discrimination, r(57) = .479, p < .001. The data suggests that the self generation responses are predictive of their performance on other tests. Participants performed better on the definition generation tests when they produced sentences with more correctly usage of the target words and when the sentences better conveyed the meaning of the word.

Delayed Testing Data

The delayed self generations responses from study 2 were coded using the same method as the immediate data. To determine if there were still differences between the performance on familiar words versus unfamiliar words a t-test was run. The t-test revealed that participants performed better on words that were familiar than unfamiliar words. However the average for the delayed tests for word use and meaningfulness declined from the immediate testing period ( see Table 1).


Word use scores significantly correlated with definition generation scores, r(55) = .588, p < .001, and related word scores, r(54) = .378, p = .004. Meaningfulness scores significantly correlated with definition generation scores, r(55) = .606, p < .001, and related word scores, r(54) = .377, p =.004.


Bolger, D. J., Balass, M., Landen, E., & Perfetti, C. A. (2008). Context variation and definitions in learning the meanings of words: An instance-based learning approach. Discourse Processes, 45, 122-159.

Chi, M.T.H. (1994). Analyzing the content of verbal data to represent knowledge: A practical guide Submitted to . Journal of the Learning Sciences.

Collins-Thompson, K. & Callan, J. (2005). Predicting reading difficulty with statistical language models. Journal of the American Society for Information Science and Technology, 56, 1448-1462.

Koedinger, K. R., Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19, 239-264.

Laufer, B. (2003). Vocabulary acquisition in a second language: Do learners really acquire most vocabulary by reading? Some empirical evidence. The Canadian Modern Language Review, 59, 567-587.

McKeown, M.G. (1993). Creating effective definitions for young word learners. Reading Research Quarterly 28, 1: 17-31.

Nation, I.S.P. (2001). Learning Vocabulary in Another Language. Cambridge: Cambridge University Press.

Nist, S.L. & Olejnik, S. (1995). The role of context and dictionary definitions on varying levels of word knowledge. Reading Research Quarterly 30(2), 172-193.

Pichette, F., 2005, Time spent on reading and reading comprehension in second language learning, Canadian Modern Language Review, 62(2), p. 243-262.

Roy, M. & Chi, M.T.H. (2005). Self-explanation in a multi-media context. In R. Mayer (Ed.), Cambridge Handbook of Multimedia Learning (pp. 271-286). Cambridge Press.

Schmitt, N. 2008. Instructed second language vocabulary learning. Language Teaching Research, 12, 329–63.

Scott, J.A., & Nagy, W.E. (1997). Understanding the Definitions of Unfamiliar Verbs. Reading Research Quarterly, 32(2), 184–200.