Roll - Inquiry

Helping Students Become Better Scientists Using Structured Inquiry Tasks

Summary Table

Study #	Focus	Start Date	End Date	LearnLab Site	# of Students	Total Participant Hours	DataShop?
1	Ethnography	1/2010	4/2010	UBC Physics	14	28	No, ethnographic data
2	SRL scaffold	3/2010	3/2010	UBC Physics	130	130	No, paper data
3	SRL training	1/2011	4/2011	UBC Physics	150	1200	Yes

Abstract

Scientific inquiry tasks have the potential to help students acquire deep understanding of domain knowledge, as well as improve their scientific reasoning skills. This project investigates scientific reasoning behaviors within one type of inquiry tasks - structured invention tasks. The project uses qualitative and quantitative methods to answer three questions: 1. What scientific reasoning skills are being practiced during structured invention tasks? 2. How can these skills be supported and improved? 3. Do the improved reasoning skills transfer to new domains?

Background & Significance

Scientific reasoning skills consist an important class of SRL behaviors. While traditional inquiry tasks have inherent benefits of letting students practice key self-regulatory skills, they were shown to be inefficient, and often unproductive, means of instruction. In the absence of adequate support, students often flounder and are lost within the infinite range of possibilities (Veermans, de Jong & van Joolingen, 2000). Consequently, students often fail to learn the target concepts, or at least do not learn them as efficiently as with direct instruction (Kirschner, Sweller & Clark, 2006).

This project focuses on scientific reasoning behaviors during inquiry, and studies the relationships between scientific reasoning behavior and domain learning and motivation. We focus on the Invention as Preparation for Learning framework (IPL; Schwartz & Taylor, 2004; Roll, Aleven & Koedinger, 2009). In IPL students are asked to invent novel mathematical procedures prior to receiving direct instruction on the canonical procedures. IPL was shown to improve students’ domain knowledge and motivation (Kapur & Lee, 2009; Roll, Aleven & Koedinger, 2009; Schwartz & Taylor, 2004). At the same time, students demonstrated poor metacognitive behavior, and lack of learning at the metacognitive level (Roll, 2009).

The current project first seeks to identify the scientific reasoning skills that are being practiced in IPL. The second stage of the project assesses the transferability of these skills (across domain topics, and along time). Last, we will investigate the effect of supporting these skills on students' domain and metacognitive learning.

Our theoretical framework with regard to metacognitive and SRL skills is detailed under Study I.

Glossary

• Structured Invention Tasks
• Self-Regulated Learning
• Scientific Reasoning
• Inquiry tasks

Research questions

1. What metacognitive processes are involved in scientific reasoning (in the context of scientific invention activities)?

2. How do different levels of support for students’ SRL affect students’ learning of domain knowledge and SRL skills?

3. How well do the acquired SRL Skills transfer across domains, tasks, and time?

Study 1: an Ethnography

The goal of Study 1 was to identify key SRL processes during the invention activity. In particular we chose to focus on skills that are crucial to the success of the invention process, and that students often skip or apply ineffectively. The study was done in two college level courses: A first year Physics lab (4 sections, 175 students), and a large first year Biology class (900 students, 3 sections). We collected over 20 hours of data from observations of groups of students working on 8 structured invention activities in these classes. In a qualitative analysis, we identified several key metacognitive behaviors that seem to contribute to the quality of students’ learning and inventions:

1. Task interpretation - Students need to read the task, and understand what is required from them. This has two components - The first is to understand what the target construct is. Students (and scientists) are expected to define what should be invented prior to attempting to invent their methods (Shute & Glaser, 1990). The second component of task interpretation (Butler & Cartier 2004) is to understand the nature of invention activity - the goal is to design a method that captures the deep invariant structure, not to embark on number hunting. Butler and Cartier (2004) emphasize task interpretation as a work habit, and so do we.

2. Qualitative analysis / goal setting – Students should interpret existing data and identify requirements that their solution should satisfy (e.g., variability should control for sample size; best fitting lines should take into account uncertainty of measurements; etc). The qualitative analysis informs the later design. It helps the students choose the right tools and activate the right prior knowledge.

3. Planning – Inventing methods to capture deep properties is a complex process. Students often ignore the goals they have set for themselves in the hustle-and-bustle of the technical invention and implementation work. The goal of the planning phase is to make sure that their goals are being met. This is done by assigning procedural components to conceptual goals (for example, controlling for sample size can be done by dividing by N).

4. Implementation - Students then need to carry out their plan. To our great surprise, we often find mismatch between students' general invented methods and their actual implementation in Excel. Yet, this phase of the task is more relevant at the domain level and is not the in the focus of our project.

5. Debugging – Upon completion of the invention process students (and scientists alike) are expected to revisit their invented method and its outcomes, and contrast them with the given data and identified goals. Does their method achieve its purpose? If not, students are expected to revise their invented method. This process includes verifications (Schoenfeld 1987) and adaptations (Winne & Hadwin, 1998).

Conceptual Framework

Our conceptual framework is based upon those put forward by Winne and Hadwin (1998) and Schoenfeld (1987)

Winne & Hadwin, 1998	Schoenfeld, 1987	Roll et al
Task definition	Read	Task interpretation
Goals & Plans	Analyze	Qualitative analysis
	Plan	Plan
Studying tactics	Implement	Implement
	Explore
Adaptations	Verifying	Debugging

Study 2: SRL prompts

The goal of Study 2 was to evaluate whether metacognitive prompts help students apply the desired SRL strategies, and whether improvement in SRL, in turn, leads to improved performance at the domain level (i.e., invention of more sophisticated and accurate methods). The study was an in vivo study with 175 students and included two conditions. All of the students received identical data (the cover story was about a new engine that was being tested). Students were asked to invent a method for evaluating the uncertainty of the slope of the linear regression of the data. In addition to domain-level information, half of the students received prompts that encouraged them to (a) analyze the data and identify goals for their method (and by that to promote goal setting) (b) compare their analyses with their peers (and by that to promote planning), and (c) evaluate the success of their invented methods (and by that to promote verifying).

Independent Variables

SRL prompts: with (treatment) or without (control).

Dependent Variables

Performance on invention tasks

Results

We found that students who received prompts were more likely to incorporate key features in their methods. Also, while most students were found to spontaneously add some form of comments to their invented methods (no self-explanation prompts were given), those who received metacognitive prompts were more likely to add higher quality self-explanations. That is, their comments discussed key features of the domain (rather than lower-level technical aspects), describing goals for their invention or evaluations of their methods. These results suggest that metacognitive prompts help students apply SRL strategies and reach better results at the domain level. However, we are yet to find out whether such instruction helps students learn better over time – that is, will students acquire better SRL skills given SRL support, and will these skills transfer to new tasks at the domain level?

Study 3 (planned): SRL training

Our goals for year 7 are to develop the Invention Lab 2.0, a system capable of facilitating multiple invention activities, accept and analyze sophisticated mathematical models, and offer different levels of metacognitive scaffolding and feedback. This system will be used to test whether the metacognitive support helps students internalize the targeted metacognitive skills, i.e., become better scientists. An in vivo study will compare 3 conditions: high metacognitive scaffolding, low metacognitive scaffolding, and fading of metacognitive scaffolding. In addition, all conditions will receive metacognitive instruction related to the relevant metacognitive strategies. The experiment will include 5 activities delivered over a period of 4 months. We will create instruments to directly assess students' metacognitive behaviors. We will evaluate whether students become better independent learners, as well as improve learning at the domain level. The transfer task includes an unsupported structured inquiry task that evaluates students' ability to learn, independently, from common scientific resources.

Independent Variables

Three level of SRL support:

• Hi scaffold
• Low scaffold
• Fading scaffold

Dependent variables

• SRL behavior
• Performance on invention tasks
• Independent performance on future assessments

Hypothesis

1. Support for students' SRL behavior will improve performance in invention activity 2. Support for students' SRL behavior will improve learning (that is, independent performance on future tasks) 3. A combination of fading feedback and direct instruction will lead to robust improvement to students' SRL skills

Results

study will be conducted during Spring 2011.

Connections to Other Studies

This is part 2 of the IPL study previously completed by Roll.

It is also related to the study of Dan Belenky and Tim Nokes about motivational benefits of IPL.

References

Butler, D. L., & Cartier, S. C. (2004). Promoting effective task interpretation as an important work habit: A key to successful teaching and learning. Teachers College Record, 106(9), 1729-1758.

Kapur, M., & Lee, K. (2009). Designing for productive failure in mathematical problem solving. In Proceedings of the 31st annual conference of the cognitive science society. (pp. 2632-7). Austin, TX: Cognitive Science Society.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86.

Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

Roll, I., Aleven, V., & Koedinger, K. R. (2009). Helping students know 'further' - increasing the flexibility of students' knowledge using symbolic invention tasks. In N. A. Taatgen, & H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society. (pp. 1169-74). Austin, TX: Cognitive Science Society.

Schoenfeld, A. H. (1987). What's all the fuss about metacognition? In Cognitive science and mathematics education. (pp. 189-215). Hillsdale, N.J: Erlbaum.

Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.

Veermans, K., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation-based discovery learning environments through intelligent support. Interactive Learning Environments, 8(3), 229-255.

Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. Graesser (Eds.), Metacognition in educational theory and practice. (pp. 277-304). Hillsdale, NJ: Erlbaum.