Fluency Summer Intern Project 2011

Background

Lexical richness plays a role in terms of overall language proficiency. Lexical richness is a broad term encompassing the depth and breadth of vocabulary a speaker is able to deploy.

Several papers have proposed measures of lexical richness, which all measure different aspects of lexical richness. Read (2000) unpacks the idea of lexical richness into lexical diversity and lexical sophistication. Lexical diversity measures range of words used by a speaker, whereas lexical sophistication measures the amount of advanced words a speaker uses. Measures of richness usually focus on either diversity or sophistication.

Measures of lexical sophistication depend on using a word list to define ‘advanced words’. Different studies have used different word lists. For instance, the Academic Word List (Coxhead, 2000) collected classroom data (van Hout & Vermeer 1992), the spoken segment of BNC corpus (Skehan 2009). Diverging approaches pose problems to the design of studies. Skehan (2009) has done a similar study comparing measures of lexical diversity and lexical sophistication. Skehan finds that measures of diversity and sophistication are very poorly correlated, and posits that these two aspects of lexical richness are independent. Given a range of task types, Skehan argues that lexical sophistication is related to the conceptualizer and diversity is related to the formulator in Levelt’s (1989) model of L2 production.

Research Question

Given several measures of lexical richness, which perform the best at discriminating group differences between EFL learners at low-intermediate, high-intermediate and low-expert levels? Given different word lists, which perform the best for measures of lexical sophistication?

How are lexical diversity and lexical sophistication related? Do they exist as independent constructs, as in Skehan (2009) or are they related? Is their relation related to proficiency levels?

Measures

Several measures exist for evaluating lexical richness, some which are more focused on diversity and others which are more focused on sophistication. Below is a brief outline of the measures used in the project. Most of these measures are not well suited to short transcripts (> 100 words).

Lexical Diversity

Lexical diversity is most basically measured by the type-token ratio, which looks at the number of types (lexemes) to the number of tokens (words). This measure is however problematic, as the number of unique types in a text decreases as the text gets longer. Each of the following measures offers a compensation for text length.

Measures

• G (Guiraud, 1954)

Simple re-calculation of TTR by using Types/√ Tokens.

• voc-D (Malvern & Richards, 1997)

Measure of vocabulary deployment. Computed by using a curve fitting operation on the TTR of random samples of tokens of different sizes. Shown to be a robust measure of lexical diversity. voc-D is used as the standard measure of lexical diversity in EFL studies.

• MTLD (McCarthy, 2006)

The average number of tokens to reach a TTR criteria (0.72). MTLD is computed sequentially forwards and backwards. MTLD has been used in validation studies and written corpora and seems to be a promising measure of lexical diversity.

Lexical Sophistication

Lexical sophistication is the ability of a speaker to be able to deploy advanced words. Studies differ in how they define 'advanced words'. This study uses three different word lists to operationalize 'advanced' words.

1. BNC Spoken Segment (10 million tokens), Cutoff 150/million
2. BNC Spoken Segment (10 million tokens), Cutoff 150/million + Task Specific words (words used by more than half the participants in a task.
3. ELI Corpus (50,000 tokens). Data collected across all transcripts used in the study. Cutoff 140/million.

Measures

• Advanced G (Daller et. al, 2003)

Modified version of the Guiraud measure: Advanced Types/√ Tokens. Measures the range of advanced types used.

• P_Lex (Meara & Bell, 2011)

Sophisticated measure which counts the number of advanced words occuring every 10 words, and fits a poisson distribution to the count frequencies. Measures the saturation of advanced words in text.

Participants & Data

Data was collected through the English Language Institute of the University of Pittsburgh. Participants (n=85) recorded 2-minute speeches (n=345) on different topics as part of the teaching curriculum of a spoken English courses. Speeches were recorded at three different times during the semester. Participants were in two cohorts, and speeches were collected across three semesters and three levels (3,4,5) of English Speaking classes.

Participants come from mixed L1 backgrounds, and most participants have significant experience studying English in a classroom prior to the ELI course (71% studied English > 3 years).

Analysis & Results

All speeches from each level was grouped together, one-way ANOVA was used to estimate effect sizes.

Effect Sizes

Measure	Effect Size (eta squared)
MTLD	0.122
G	0.103
D	0.076
L3_AG	0.146
L1_AG	0.089
L2_AG	0.065
L2_PLex	0.060
L1_PLex	0.059
L3_PLex	0.020

• Measures of lexical diversity are robustly able to discriminate groups. MTLD has the highest effect size of lexical diversity measures.
• Lexical sophistication measures (using list 1 & 2) are poorly able to discriminate group differences. Participants near-equal levels of sophistication scores can be explained by prior English education (focusing on vocabulary).
• Advanced Guiraud using word list 3 performs the best at discriminating group differences. However, word list 3 is biased in it's construction (topics were not balanced by number of participants).
• Topics varied in complexity and accordingly there was a significant interaction of measures of lexical diversity and Advanced Guiraud within-levels between topics in levels 3,4.
• P_Lex is an inappropriate measure to use for short texts, was unable to discriminate group differences.

Diversity & Sophistication

• Skehan's findings replicated: moderate correlations between lexical diversity measures and lexical sophistication measures. (0.41 correlation between C3_AG and MTLD).

To check construct validity, factor analysis was run using Principal Components Analysis. Using z-normailzed measures of G,D,MTLD and Advanced Guiraud measurements using all three word lists factor analysis was performed. Two components were identified (λ > 1.0). Promax rotation with kaiser normalization was performed on the principal components.

We can conclude based on the moderate correlation and the composition of the rotated components that lexical diversity and lexical sophistication can be considered separate, but related constructs.

References

Daller, H. et al. (2003) Lexical Richness in the Spontaneous Speech of Bilinguals. Applied Linguistics (24) 2, 197-222

Guiraud, P. (1954) Les caractères statistiques du vocabulaire, Paris: Presses universitaires françaises.

Levelt, W.J.M. (1989) Speaking: from intention to articulation, Cambridge, Massachusetts: The MIT Press.

Malvern D. D. & Richards, B. J. (1997) A new measure of lexical diversity. In A. Ryan & A. Wray (Eds.), Evolving models of language 58-71 Clevedon: Multilingual Matters

McCarthy, P.M. (2006) An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD). Dissertation Abstracts International 66 (12)

Meara, P. & Bell, H. P_Lex: A simple and effective way of describing the lexical characteristics of short L2 texts. Prospect 16 (3), 5-19

Skehan, P. (2009) Modelling Second Language Performance: Integrating Complexity, Accuracy, Fluency, and Lexis. Applied Linguistics 30 (4)