Intelligent Writing Tutor

PI	Teruko Mitamura
Graduate Students	Ruth Wylie
Faculty	Ken Koedinger, Brian MacWhinney
Others with > 160 hours	Jim Rankin
Start date study 1	March 2006
End date study 1	June 2007
Learnlab	ESL
Number of students	10 (current), 100 (expected total)
Total Participant Hours	~30 (current), ~300 (expected total)
Datashop?	Yes

Intelligent Writing Tutor

Teruko Mitamura, Ruth Wylie, and Jim Rankin
Project Advisors: Brian MacWhinney and Ken Koedinger

Abstract

As part of contributing to the PSLC's underlying goal of developing a theory of robust learning, the Intelligent Writing Tutor (IWT) project will explore the issue of transfer and long-term retention of acquired knowledge. We will look at positive, negative and null transfer from a student’s native language (L1), and their roles in learning and retaining information. We hope to be able to develop general principles that can then be applied to other domains.

Background

Research Motivation

We are interested in looking at the effects of transfer from L1 in language learning and English in particular. This study will contribute to the theory of robust learning by providing experimental data related to the often cited but little researched educational principle of “build on prior knowledge”. We hypothesize that elements of English that correspond to a student's L1 will be easier to learn than those elements that do not correspond (positive transfer). Moreover, elements for which there are no corresponding elements will be harder to learn (negative transfer). Our study will go beyond the simple Contrastive Analysis by examining at a detailed leveled the various features and their relative validity for a given knowledge component. For example, instead of looking at simply if articles exist in the native language, we will examine specific instances of article usage (e.g. immediate situation, general knowledge, sporadic reference, etc. (Sand, 2004)).

Educational Motivation

While error-free article use may not be necessary in order for most communication to occur (e.g. People usually understand what is meant by “Give me a book on the table” even when “Give me the book on the table” is correct), the problem becomes more severe when students submit written work, especially high-level work such as technical and academic reports. Errors in writing at this level may suggest, either consciously or unconsciously, that errors exist in the work itself (Master, 1997). Furthermore, students at this level are often aware of their difficulties with article use and are motivated to learn how to use articles correctly, thus developing a tutoring system that will help them develop better understanding and therefore produce error-free text and speech is likely to be well-received.

Research question

How is robust learning of English affected by L1 transfer?

Independent variables

Study 1:

The independent variable for this study is the student’s first language (L1). Students will be divided into groups based on the type of determiners present in their L1. Students with L1s do not have an article system (e.g. Chinese, Japanese, Korean) will be placed in one group, while students with L1s that do have an article system (e.g. Spanish, Arabic, French) will be placed in another.

Hypothesis

Study 1:

Hypothesis 1: Knowledge components subject to negative transfer will be harder to learn than knowledge components for which there is positive or no transfer

• Pcno – Probability of correct given no transfer
• Pc- – Probability of correct given negative transfer

Hypothesis 2: Knowledge components subject to negative transfer will exhibit greater rate of decay than knowledge components for which there is positive or no transfer.

• Pcno – Probability of correct given no transfer
• Pc- – Probability of correct given negative transfer
• PI – Probability of Incorrect

Dependent variables

This study will utilize a series of post-tests which measure both normal and robust learning, including:

• Normal post-test, immediate: Immediately following instruction, students will complete their first post-test in order to measure the effectiveness of the training itself. These tasks will be a measure of normal learning (near transfer, immediate testing).
• Normal post-test, long-term retention: Additional post-tests will be administered 3, 10, 20, and 35 days after initial instruction. These measures will be similar to the ones students encountered during training but will assess the more robust learning measure of long-term retention.
• Transfer: In addition, we will collect student writing samples in order to determine if the instructional activities succeeded in enabling students to produce text with fewer errors.

Explanation

This study is part of the Fluency and Refinement cluster. The main hypothesis of this cluster is that the structure of instructional activities and student’s prior knowledge play critical roles in developing robust learning. Since students learning English are already fluent in at least one language, we can utilize this fact to better understand how a student’s prior knowledge affects acquisition of new knowledge components. The learning event space is described as follows:

Start

1. Guess
   1. Entry is correct --> exit, with little learning
   2. Entry is incorrect --> Start
2. Use the article of one’s first language
   1. Entry is correct --> exit, with possibly mistaken learning
   2. Entry is incorrect --> Start
3. Try to apply knowledge of English article grammar
   1. Entry is correct --> Exit, with learning
   2. Entry is incorrect --> Start

The second set of paths (2, 2.1, 2.2) are only available to students whose first language has articles. Thus, this analysis is an instance of the explanation schema adding new paths.

Although this study seems on the surface to be a simple matter of measuring negative transfer, the learning events space analysis suggests that learning is contingent on students’ choices of paths. In particular, if the grammatical system of the first language is quite different from the English system, students may rapidly learn that choice 2 leads only to errors (2.2) so they may stop using their first language as the default solution. In that case, the expected negative transfer may not occur.

The study includes retention and acceleration of future learning measures. This allows testing of the so-called path independence hypothesis (Klahr & Nigam; Nokes & Ohlsson), which is that when students reach a certain level of competence, it doesn’t matter how they got there; their subsequent performance, including both retention and acceleration, will be the same. The PSLC theoretical framework suggests that the path to competence does make a difference, albeit a small one. If students make several errors per learning event, and thus have to cycle through the paths above several times, then when they do eventually produce the correct response, the encoding context is cluttered with features due to the errors and feedback messages. During testing, those features will be absent. Thus, students may be less able to retrieve the appropriate knowledge components. This predicts that students who make multiple errors during training, and this is likely to be the students who have path 2 available to them, are likely to have less robust learning.

Pilot Study

Think-Aloud Study

A study was conducted in order to examine differences between the two task types as well as gather verbal protocols in order to understand which rules and heuristics students employ when solving these problems. If students performed equally well (or poorly) using the different interfaces, one would not expect to see differences in learning. In answering these questions, we performed a think-aloud study in which we invited ELL participants into the lab and asked them to complete tasks using the two systems. Students were told beforehand that the only errors present within the paragraphs were article errors.

Task Content

The problem paragraphs came from intermediate and advanced-level ESL textbooks (e.g. [2], [3]). The first problem was shorter in length and used simpler vocabulary than the second problem did. Students were randomly assigned to one of two groups; those in Group 1 completed the intermediate level problem using the editing (production and detection) interface and the advanced level problem with the menu (production-only) interface. Students in Group 2 did the opposite (See Table 1).

Table 1. Type of interface used for each problem

	Intermediate Level Problem	Advanced Level Problem
Group 1	Production + Detection Task	Production Only Task
Group 2	Production Only Task	Production + Detection Task

Participants

Participants were recruited from Carnegie Mellon University’s InterCultural Communication Center’s Academic Culture and Communication (ACC) program. The program is a six week summer program designed for newly admitted nonnative English speaking students. Traditionally, students entering the program have high TOEFL scores (greater than 580) and at least an intermediate level of spoken fluency. ACC students are a highly educated and highly motivated group.

In total, there were six participants (three female, three male), and all were native Chinese speakers. The average age was 28 years old and all students had been learning English since middle school (average number of years = 14.4 years). Using self-report scales, participants gave an average proficiency score of 3.4 for reading, 3.0 for writing, and 2.5 for speaking (where 1 represents absolute beginner and 5 native or fluent).

The participants of this study represent a population that is perhaps more advanced than the overall target population of these systems. This is due to the think-aloud methodology employed for the pilot study. We needed students whose English ability was high enough that they were able to verbalize what they were doing. However, because English articles are often one of the last grammar points for students to master, we were able to use texts that were challenging even for these advanced participants.

Results

The data reveal a strong performance difference between the two tasks. Again, the difference in the two interfaces was that in the production-only version, students only had to choose which article to insert for each given box. In the production and detection condition, students had to identify and correct the errors. If students were able to correctly locate errors but had difficulty correcting them, we would expect for the results to be the same regardless of the interface. However, if error identification is the true obstacle, we would expect students in the detection and correction condition to perform worse than students using the production-only tutor.

For both problems, the production-only task resulted in higher accuracy, as measured by the percent of necessary changes that were correctly made. While the difference between the two interfaces was not significantly different for the easier, intermediate-level problem (t=1.45, p = 0.13), the performance results were significantly different for the advanced problem (t=2.13, p = 0.05), suggesting that when students are presented with level-appropriate texts, detecting errors is a formidable challenge.