Physics

Physics LearnLab Course

Studies conducted using Andes.

Use last year's report as starting point.

• need participant hours from expt. pages
• need objectives for coming year.
• need number of experiments planned/run

Old Report

The Physics LearnLab Course The Physics LearnLab Course (PLLC) provides a research facility for studying how students learn physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the General Physics I and II classes at the US Naval Academy in Annapolis, MD. In subsequent years, it will expand to include more sites. Students in PLLC classes use the Andes intelligent tutoring system (http://www.andes.pitt.edu) in order to do their homework. Andes not only helps students learn physics, as demonstrated in earlier evaluations, but it also allows the PLLC to collect fine-grained data on student activities while solving their homework problems. Almost everything in the course is taught in the usual way, with lectures, labs, and a commercial paper-based textbook. In vivo experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

I. PLLC Plan: Current state: The PLLC is currently comprised of 3-5 sections (depending on the experiment) of 25 students each. The sections are taught by US Naval Academy Professors Don Treacy, Mary Wintersgill and John Fontinella. The students use OLI to access Andes, and the instructors use OLI to view gradebooks. Raw log data from Andes is captured by OLI, but the conversion to Datashop format is only partially completed. Other data, such as hardcopies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments and stored in locked file cabinets or secure servers. All data use the USNA “alpha numbers” to label students; only the instructors and the USNA administration have access to the mapping between alpha numbers and students’ identities. Long term plan: Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. Also, several of the instructors currently involved are nearing retirement age. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses. So far, we have had limited success in doing this. Instructors have been reluctant to adopt Andes for the following reasons: instructors want a homework system to cover an entire course; they want any hints given to be effective; and they want reasonable student actions to be accepted. Finally, Andes is not well-known in the general physics community. During the first two years of the PLLC (Jan 05 to December 06), we have invested almost all our effort to giving Andes full course coverage by adding new physics knowledge and problems to Andes. Andes now covers 100% of the Fall semester, and 75% of the Spring semester homework in the sections taught by Treacy and Wintersgill at the US Naval Academy. The remaining problems involve calculus notation and cannot easily be implemented in Andes Because we largely achieved the first long-term goal of complete course coverage, we are shifting the focus of our attention to increasing the awareness of Andes in the physics teaching community, increase the quality of hints provided in Andes, improve our quality control process, and provide additional course content requested by new physics instructors.

Achievements: From its inception in January 2005 to the present report written in December 2006, we have achieved the following:

(1) Content development milestones: The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester. We have increased the total number of working Andes problems from 350 to 546. The number of physics principles has increased from 126 to 232. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 927. The number of scalar quantities defined in Andes has increased from 85 to 113. We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

(2) Enabling Technologies: We developed a way to run Andes under OLI. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes. We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified. Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.

(3) Advertising Andes in the physics community Presented Andes at the American Association of Physics Teachers (AAPT) conference in Syracuse, NY, July 2006. (one talk and 3 poster sessions). Presented a talk on Andes at Southern Methodist University, Nov. 2006 Presented a talk on Andes at Ohio State Unviersity, Nov. 2006.

(4) Adoption of Andes at new institutions As of fall 2006, Andes is being used at the following institutions (one section, one instructor each): St. Anselm college, Manchester NH Dominican College, Troy NY Anne Arundel Community College, Arnold, MD US Air Force Academy

(5) Journal and conferences presentations: VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. International Journal of Artificial Intelligence and Education, 15 (3). Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. The Artificial Intelligence in Education Conference, Amsterdam, 2005. VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R. G. M., Weinstein, A., et al. What's in a step? Toward general abstract representations of tutoring system log data. Paper submitted to the 11th International Conference on User Modeling, Corfu, Greece.

(6) Publications: VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. (2005). The Andes Physics Tutoring System: Lessons Learned. International Journal of Artificial Intelligence and Education, 15 (3). Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. (2005). The Andes physics tutoring system: Five years of evaluations. In: G. I. McCalla and C.-K. Looi (Eds.), Proceedings of the Artificial Intelligence in Education Conference. Amsterdam: IOS. VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R. G. M., Weinstein, A., et al. (submitted). What's in a step? Toward general abstract representations of tutoring system log data. Paper presented at the 11th International Conference on User Modeling, Corfu, Greece 2006. Submitted to the proceedings.

Goals for the next report period Log file analysis. Andes raw logs can be converted to the DataShop format, but the converted logs do not have the right information in them for the kinds of analysis experimenters want to do. Increasing Andes awareness in the physics community Present talks and posters at the American Association of Physics Teachers conference in January 2007 and the American Physics Society (APS) meeting in March 2007. Continue to visiting physics departments at other universities. Publish Andes-based research in the physics education journals. Supporting existing Andes instructors. There are a number of non-PLLC instructors using Andes in their classrooms. We need to make their experience with Andes a positive one. Hopefully, a positive experience will lead to interest in participating in LearnLab. This includes: Adding instructor-requested homework problems Fixing instructor-reported bugs promptly, and Including some instructor control over the hinting behavior of Andes. Increasing instructor acceptance. In order to increase the number of instructors using Andes, we need to address the following issues: Improving hints given to students. Andes is supposed to mimic the hints that an expert (human) tutor would give to students. However, sometimes the hints are misleading or do not help the student better understand the problem at a deep level. Improving quality control. Andes is a rather complicated system and our current method of quality control, solving Andes problems by hand and reacting to instructor/student feedback, has proved to be insufficient. We hope to implement a new technique for testing changes to Andes using the logs of student solutions stored at OLI. Also, we plan to search the student log files for Andes errors and evidence of poor hints. Non-obvious conventions. Communicating physics and math precisely and unambigously entails using some notational and user interface conventions that user find non-obvious. We call these non-obvious conventions (NOCs; pronounced “knocks”). We need to collect as many non-obvious conventions as we can think of and fix them or highlight them in the videos and other training. Lesser priority items. There are a number of improvements to Andes that would lead to increased instructor happiness: Vectors in equations. Handling of equivalent variables. And the equations that use them. Scale drawing of vectors. True but irrelevant entries. Currently these turn red. Instructor should be able to select the color and/or the warning that appears. They should be able to have separate policies for equations and non-equations.

II. PLLC Research and Experiments:

Table:

In Vivo Pull Out Lab Capacity Course Run

Planned Run Planned Run Planned Total #Sections Total

1. Students

Max #Studies /Year Max #Students /Study

6 2 0 0 0 0 4 100

Approved project plans : (1) “The effect of generation and interaction on robust learning,” Bob Hausmann & Kurt VanLehn, Pitt. (2) “Investigating the robustness of vicarious learning,” Scotty Craig & Micki Chi, Pitt. (3) “The effectiveness of post-practice reflective dialogues,” Sandy Katz, Pitt. (4) “Scaffolding problem solving with annotated worked-out examples to promote deep learning,” Michael Ringenberg & Kurt VanLehn, Pitt.

(5) “Reflective dialogues that explicitly target the What? How? Why(not)? Of physics,” Sandy Katz, Pitt. 

(5)“Bridging principles and examples through analogy and explanation,” Tim Nokes, Pitt.

In-vivo Experiments run in the PLLC:

Spring 2005: Mike Ringenberg conducted his study on worked examples.

Fall 2005: Sandy Katz completed her first study on reflective dialogs.

Spring 2006: Bob Hausman and Scotty Craig completed their first studies.

Fall 2006: Sandy Katz completed her second study on reflective dialogs. Scotty Craig completed his second study on worked examples.

Looking Ahead for Spring 2007: Bob Hausmann is scheduled to do a second study.

Studies Conducted

• Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)

• The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)

• Post-practice reflection (Katz & Connelly, 2005)

• Deep-level questions during example studying (Craig & Chi, 2006)

• Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)

• Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)

• Reflective Dialogues (Katz, Connelly & Treacy, 2007)

• The effects of interaction on robust learning (Hausmann & VanLehn, 2007)

• Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)