Physics

From LearnLab
Revision as of 03:29, 27 April 2008 by Bvds (talk | contribs) (→‎Achievements: fix indent)
Jump to: navigation, search

Physics LearnLab Course

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the Andes intelligent tutoring system to do their homework. Andes allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. In vivo experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

Studies Conducted

Summary of Studies
In Vivo Pull Out Lab Capacity
Course Run Planned Run Planned Run Planned Total # Sections Total # Students Max # Studies / Year Max # Students / Study
Physics 9 1 0 0 3 0 5 130 4 65

Capacity was determined by counting the number of students who solved more than 40 Andes problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

In progress or planned:

Achievements

From its inception in January 2005 to the present, we have achieved the following:

  • Content development milestones
    • The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
    • We have increased the total number of working Andes problems from 350 to 556.
    • The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
    • We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.
  • Enabling Technologies
    • We developed a way to run Andes under OLI. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
    • We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
    • Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
    • Andes raw logs can now be converted to the [DataShop] format at the knowledge component level (June 2007). The knowledge components associated with each correct student action (corresponding with a step) and must incorrect action (see transaction) is determined by Andes.
  • Adoption of Andes As of spring 2008, Andes is being used at the following institutions:
    • St. Anselm college, Manchester NH (1 instructor).
    • US Naval Academy (2 instructors, several sections).
    • Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
    We observe steadily growing use of Andes by individuals not enrolled in any OLI course. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.
  • Pulblications on Andes:
    • VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. International Journal of Artificial Intelligence and Education, 15 (3), 1-47.
    • Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), Artificial Intelligence in Education. (pp. 678-685) Amsterdam, Netherlands: IOS Press.
    • Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp 246-253. Amsterdam, Netherlands: IOS Press.
    • VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). Proceedings of User Modelling 2007.
    • VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.
  • Publications on PLLC experiments:
    • Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. Technology, Instruction, Cognition, and Learning, 4, 1-29.
    • Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), Intelligent Tutoring Systems: 8th International Conference, ITS2006. pp. 625-634. Amsterdam: IOS Press.
    • Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial Intelligence in Education 2007.
    • Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) Proceedings of the 29th Annual Conference of the Cognitive Science Society. pp. 167-172 Mahwah, NJ: Erlbaum.
    • Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 548-550. Amsterdam, Netherlands: IOS Press.
    • Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 551-553. Amsterdam, Netherlands: IOS Press.
    • Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 409-416. Amsterdam, Netherlands: IOS Press.
    • Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 554-556. Amsterdam, Netherlands: IOS Press.
    • Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 417-424. Amsterdam, Netherlands: IOS Press.
    • Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) Proceedings of the 29th Annual Conference of the Cognitive Science Society. pp 1067-1072 Mahwah, NJ: Erlbaum.
    • Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
    • Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
    • Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

Current Status

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The students use Open Learning Initiative (OLI) to access Andes, and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to DataShop format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

Plans

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In addition, our most supportive instructor at the US Naval Academy, Don Treacy, has recently retired. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses. So far, we have had limited success in doing this. Instructors have been reluctant to adopt Andes for the following reasons: instructors want a homework system to cover their entire course; they want Andes problems to fit their instructional style; they want any hints given to be effective; and they want reasonable student actions to be accepted. Finally, Andes is not well-known in the general physics community.


Because we largely achieved the first long-term goal of complete course coverage, we are shifting the focus of our attention to increasing the awareness of Andes in the physics teaching community, increase the quality of hints provided in Andes, improve our quality control process, and provide additional course content requested by new physics instructors.


  • Increasing Andes awareness in the physics community
    • Present talks and posters at the American Association of Physics Teachers conference in January 2008 and the American Physics Society (APS) meeting in April 2008.
    • Continue to visiting physics departments at other universities.
    • Publish Andes-based research in the physics education journals.
  • Supporting existing Andes instructors: There are a number of non-PLLC instructors using Andes in their classrooms. We need to make their experience with Andes a positive one. Hopefully, a positive experience will lead to interest in participating in LearnLab. This includes:
    • Adding instructor-requested homework problems
    • Fixing instructor-reported bugs promptly, and
    • Including some instructor control over the hinting behavior of Andes.
  • Increasing instructor acceptance: In order to increase the number of instructors using Andes, we need to address the following issues:
    • Improving hints given to students: Andes is supposed to mimic the hints that an expert (human) tutor would give to students. However, sometimes the hints are misleading or do not help the student better understand the problem at a deep level.
    • Non-obvious conventions: Communicating physics and math precisely and unambigously entails using some notational and user interface conventions that user find non-obvious. We call these non-obvious conventions (NOCs; pronounced “knocks”). We need to collect as many non-obvious conventions as we can think of and fix them or highlight them in the videos and other training.
    • Lesser priority items: There are a number of improvements to Andes that would lead to increased instructor happiness.
      • Vectors in equations: Handling of equivalent variables. And the equations that use them.
      • Scale drawing of vectors.
      • True but irrelevant entries: Currently these turn red. Instructor should be able to select the color and/or the warning that appears. They should be able to have separate policies for equations and non-equations.
Retrieved from "https://learnlab.org/wiki/index.php?title=Physics&oldid=7978"