Nokes - Questionnaires

Summary Table

PIs	Timothy Nokes, Vincent Aleven
Other Contributors	Matthew Bernacki
Study Start Date	Study 1 (Greenville) Oct 2010 Study 2 (Hopewell)- August 2011
Study End Date	Data collections are complete, analyses are ongoing.
Site	Greenville High School 2010-11; Hopewell SD 2011-12
Number of Students	67 Greenville Geometry students; ~300-350 Hopewell students (~125 per Bridge/Algebra/Geometry)
Total Participant Hours
Data available in DataShop	Dataset: Affect Detectors and Questionnaires Greenville 2010-11 Pre/Post Test Score Data: TBD Paper or Online Tests: TBD Scanned Paper Tests: TBD Blank Tests: TBD Answer Keys: TBD

Abstract

We will develop infrastructure to collect variables related to metacognition, affect, and motivation across all LearnLabs. The planned LearnLab instrumentation involves two innovations in measurement: We will use microgenetic approaches for the fine-grained sampling of constructs vis-à-vis 1) the repeated administering of brief questionnaire items and less frequent longitudinal sampling using longer questionnaires, and 2) moment-by-moment behavioral data, including automatic monitoring in learning software. Our unique strength in these areas will be the ability to combine rich layers of behavioral measures (cognition, metacognition, affect, motivation), which will be used to create online models that can predict moment-by-moment changes. In doing so, we will leverage DataShop capabilities; the DataShop has been designed explicitly to accommodate multiple interpretations of student interaction data, if necessary at different grain sizes.

This project will enable us to collect data on metacognitive, motivational and affective states in naturalistic learning settings at an unprecedented level of fine-grained detail (both temporal as well as type, multiple simultaneous measures). This level of detail will enable the PSLC learning scientists and learning scientists at large (through the DataShop) to test novel questions and theoretical models of the relation between M&M behaviors and states and robust learning that have been previously unable to be tested. Furthermore, this project will enable us to test the generalizability of current theoretical models in the literature (e.g., Blackwell, Trzesniewski, & Dweck, 2007).

Plans to Assess the Relationship Between Motivation and Affect on Robust Learning

We will collect questionnaire data for a range of variables. This effort will have two components. First, we will take a microgenetic approach to collect questionnaire data with a small number of items that are administered frequently (i.e., dense data collection over a range of time periods, providing motivational / affective tracking from minutes to hours to weeks). These questionnaires will be embedded in the learning software and therefore can be administered between problems, or at beginning or end of session (and perhaps, subject to these constraints, randomly). This method of data collection will be applied to affective or motivational variables that are expected to vary more rapidly (e.g., interest, strategies, goal orientation towards the task, attitudes towards the learning materials). This approach will provide very fine-grained data as to how motivation and affective states change based on changes in the learning environment or task structure (e.g., difficulty, problem type, topic, domain, etc.), as well as student interaction with the tutor or peers (e.g., strategies, cognitive processing, etc.).

Second, twice or three times a year we will administer questionnaires focused on constructs that may be semi-stable over time (e.g., self-efficacy, attitudes towards the domain, theory of intelligence, goal orientation towards the domain), a very traditional method in motivational research or research in SRL, although one whose shortcoming are increasingly being noticed (Zimmerman, 2008). Key to the current approach is that this more traditional type of data can be related to fine-grained data on PSLC measures of robust learning (instead of only using grades as a measure of learning which is typically the measure used in the literature in naturalistic learning settings). These questionnaires will be administered electronically via the tutor in units explicitly designed for questionnaire administration.

The data from these two approaches will enable the Metacogniton and Motivation thrust to test path and structural equation models of the relation of particular M&M states and behaviors to robust learning (see Blackwell, Trzesniewski, and Dweck, 2007 for an example). Critically we will be linking motivational and affect variables to cognitive processes (by which they are hypothesized to do their work) and robust learning outcome measures. One goal of this work is theoretical integration of past work at the PSLC on instructional principles (macro-level), cognitive processes / knowledge components (mirco-level) and measures of robust learning to research / work and results on motivation and affect. Furthermore, this project provides a unique opportunity to test the generalizability of current and new theories of learning and motivation and affect across a number of academic domains (LearnLabs). This project will also play a critical role in the Theoretical Integration project of the thrust described in section 1.3.

These measures will be used in the experiments designed in the Social Communicative and Cognitive Factors Thrusts to provide across thrust integration. In addition, the collected data will be used to build and validate automated detectors for important aspects of students’ metacognition (described in the next section).

Our strategy will be initially to focus on a small set of variables that both builds on prior work conducted at the PSLC and the literature has identified as particularly relevant for learning in academic contexts.

• Awareness and use of SRL strategies (e.g., Motivated Strategies for Learning Questionnaire or its descendants) (Pintrich & de Groot, 1990)
• Self-efficacy (Bandura , 1997)
• Theory of intelligence (entity, incremental) (Dweck, 2006)
• Achievement goals (mastery, performance-approach, performance-avoidance, learning) (Darnon, Butera, & Harackiewicz, 2007; Elliot & Dweck, 1988; Elliot & Murayama, 2008)
• Interest (Hidi & Renninger, 2006; Linnenbrink-Garcia,Durik, Conley, Barron, Tauer, Karabenick, & Harackiewicz, 2010)

Results

Forthcoming

Explanation

Forthcoming