Theory Wiki - User contributions [en]

Physics

2011-09-06T21:07:42Z

Bvds: /* Plans */ Update to match new version of Andes

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time (2010), it is sited in:
* the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
* three courses at Watchung Hills Regional High School in Warren, NJ.
* one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

*Comparing two homework systems, Sophia Gershman, 2008-2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2009)

*An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System. Data mining study (Kasia Muldner, Winslow Burleson, Brett Van de Sande, Kurt VanLehn, 2009-2010).

In progress or planned:

*Comparing two homework systems, Sophia Gershman, 2010-2011.

*Reflective dialogs (Sandy Katz, Diane Litman, & Pam Jordan, 2010-2013).

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Spring 2010, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* Upper St. Clair School District, PA (1 instructor).
* Clark County School District, NV (1 instructor, distance learning).
* Clark College, WA (1 instructor, several sections)
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Spring 2010, between 285 and 1332 different users (some use is anonymous, precluding an exact count) solved a total of 15192 Andes problems. The previous semester, a total of 11819 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
* K. Muldner, W. Burleson, B. van de Sande, & K. VanLehn, Fun and Gaming with Andes. Poster presented at the [http://www.aapt.org/Conferences/sm2010/ 2010 AAPT Summer meeting], Portland Oregon, July 2010.
* B. van de Sande. Experimental methods for studying student Metacognition and Affect. Workshop presented at the [http://www.compadre.org/per/conferences/2010/ ''Physics Education Research Conference 2010''], Portland OR, July 2010.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Gaming Behaviors in an Intelligent Tutoring System. Proceedings of the International Conference on Intelligent Tutoring Systems, 2010, Pittsburgh, 184-193. [http://www.springerlink.com/content/92mru151006571q3/]
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System: Predictors and Impacts. Journal of User Modeling and User Adapted Interaction, Special Issue on Educational Data Mining. [http://www.springerlink.com/content/j33110w75053463l/]

==Current Status==

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally
teaches one section.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal is to expand the number of instructors and researcher involved with the PLLC.
In order for this to happen, we need to increase the number of instructors using Andes.
Also, we believe there is great mutual benifit in working closely with members of the [http://www.compadre.org/per/ Physics Education Research] community.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. Currently, Andes is available thought the open source Learning Management System [http://www.lon-capa.org LON-CAPA]. We continue to work on our goal of offering Andes through our commercial partner, [http://www.webassign.net WebAssign].
* Instructors want all reasonable student actions to be accepted. We need to expand the ability of Andes to understand a wide variety of problem solutions.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2011-02-20T05:23:50Z

Bvds: /* Advertising Andes in the physics community */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time (2010), it is sited in:
* the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
* three courses at Watchung Hills Regional High School in Warren, NJ.
* one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

*Comparing two homework systems, Sophia Gershman, 2008-2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2009)

*An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System. Data mining study (Kasia Muldner, Winslow Burleson, Brett Van de Sande, Kurt VanLehn, 2009-2010).

In progress or planned:

*Comparing two homework systems, Sophia Gershman, 2010-2011.

*Reflective dialogs (Sandy Katz, Diane Litman, & Pam Jordan, 2010-2013).

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Spring 2010, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* Upper St. Clair School District, PA (1 instructor).
* Clark County School District, NV (1 instructor, distance learning).
* Clark College, WA (1 instructor, several sections)
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Spring 2010, between 285 and 1332 different users (some use is anonymous, precluding an exact count) solved a total of 15192 Andes problems. The previous semester, a total of 11819 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
* K. Muldner, W. Burleson, B. van de Sande, & K. VanLehn, Fun and Gaming with Andes. Poster presented at the [http://www.aapt.org/Conferences/sm2010/ 2010 AAPT Summer meeting], Portland Oregon, July 2010.
* B. van de Sande. Experimental methods for studying student Metacognition and Affect. Workshop presented at the [http://www.compadre.org/per/conferences/2010/ ''Physics Education Research Conference 2010''], Portland OR, July 2010.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Gaming Behaviors in an Intelligent Tutoring System. Proceedings of the International Conference on Intelligent Tutoring Systems, 2010, Pittsburgh, 184-193. [http://www.springerlink.com/content/92mru151006571q3/]
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System: Predictors and Impacts. Journal of User Modeling and User Adapted Interaction, Special Issue on Educational Data Mining. [http://www.springerlink.com/content/j33110w75053463l/]

==Current Status==

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally
teaches one section.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2011-02-20T05:02:11Z

Bvds: Update 2010 publications

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time (2010), it is sited in:
* the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
* three courses at Watchung Hills Regional High School in Warren, NJ.
* one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

*Comparing two homework systems, Sophia Gershman, 2008-2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2009)

*An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System. Data mining study (Kasia Muldner, Winslow Burleson, Brett Van de Sande, Kurt VanLehn, 2009-2010).

In progress or planned:

*Comparing two homework systems, Sophia Gershman, 2010-2011.

*Reflective dialogs (Sandy Katz, Diane Litman, & Pam Jordan, 2010-2013).

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Spring 2010, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* Upper St. Clair School District, PA (1 instructor).
* Clark County School District, NV (1 instructor, distance learning).
* Clark College, WA (1 instructor, several sections)
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Spring 2010, between 285 and 1332 different users (some use is anonymous, precluding an exact count) solved a total of 15192 Andes problems. The previous semester, a total of 11819 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Gaming Behaviors in an Intelligent Tutoring System. Proceedings of the International Conference on Intelligent Tutoring Systems, 2010, Pittsburgh, 184-193. [http://www.springerlink.com/content/92mru151006571q3/]
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System: Predictors and Impacts. Journal of User Modeling and User Adapted Interaction, Special Issue on Educational Data Mining. [http://www.springerlink.com/content/j33110w75053463l/]

==Current Status==

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally
teaches one section.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2010-11-11T22:37:49Z

Bvds: /* Adoption of Andes */ Update usage using Spring 2010 data.

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time (2010), it is sited in:
* the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
* three courses at Watchung Hills Regional High School in Warren, NJ.
* one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

*Comparing two homework systems, Sophia Gershman, 2008-2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2009)

*An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System. Data mining study (Kasia Muldner, Winslow Burleson, Brett Van de Sande, Kurt VanLehn, 2009-2010).

In progress or planned:

*Comparing two homework systems, Sophia Gershman, 2010-2011.

*Reflective dialogs (Sandy Katz, Diane Litman, & Pam Jordan, 2010-2013).

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Spring 2010, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* Upper St. Clair School District, PA (1 instructor).
* Clark County School District, NV (1 instructor, distance learning).
* Clark College, WA (1 instructor, several sections)
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Spring 2010, between 285 and 1332 different users (some use is anonymous, precluding an exact count) solved a total of 15192 Andes problems. The previous semester, a total of 11819 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Gaming Behaviors in an Intelligent Tutoring System. To Appear in the International Conference on Intelligent Tutoring Systems, 2010, Pittsburgh, 10 pages.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System: Predictors and Impacts. Submitted to the
UMAISpecial Issue on Data Mining for Personalized Educational Systems.

==Current Status==

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally
teaches one section.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Nokes - Questionnaires

2010-07-21T03:41:51Z

Bvds: /* Plans to Assess the Relationship Between Motivation and Affect on Robust Learning */

==Summary Table==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Timothy Nokes, Vincent Aleven
|-
| '''Other Contributers''' || Daniel Belenky
|-
| '''Study Start Date''' ||
|-
| '''Study End Date''' ||
|-
| '''Site''' ||
|-
| '''Number of Students''' ||
|-
| '''Total Participant Hours''' ||
|-
| '''DataShop''' ||
|}
<br>

==Abstract==
We will develop infrastructure to collect variables related to [[metacognition]], affect, and motivation across all LearnLabs. The planned LearnLab instrumentation involves two innovations in measurement: We will use microgenetic approaches for the fine-grained sampling of constructs vis-à-vis 1) the repeated administering of brief questionnaire items and less frequent longitudinal sampling using longer questionnaires, and 2) moment-by-moment behavioral data, including automatic monitoring in learning software. Our unique strength in these areas will be the ability to combine rich layers of behavioral measures (cognition, metacognition, affect, motivation), which will be used to create online models that can predict moment-by-moment changes. In doing so, we will leverage DataShop capabilities; the DataShop has been designed explicitly to accommodate multiple interpretations of student interaction data, if necessary at different grain sizes.

This project will enable us to collect data on metacognitive, motivational and affective states in naturalistic learning settings at an unprecedented level of fine-grained detail (both temporal as well as type, multiple simultaneous measures). This level of detail will enable the PSLC learning scientists and learning scientists at large (through the DataShop) to test novel questions and theoretical models of the relation between M&M behaviors and states and robust learning that have been previously unable to be tested. Furthermore, this project will enable us to test the generalizability of current theoretical models in the literature (e.g., Blackwell, Trzesniewski, & Dweck, 2007).

==Plans to Assess the Relationship Between Motivation and Affect on Robust Learning==
We will collect questionnaire data for a range of variables. This effort will have two components. First, we will take a microgenetic approach to collect questionnaire data with a small number of items that are administered frequently (i.e., dense data collection over a range of time periods, providing motivational / affective tracking from minutes to hours to weeks). These questionnaires will be embedded in the learning software and therefore can be administered between problems, or at beginning or end of session (and perhaps, subject to these constraints, randomly). This method of data collection will be applied to affective or motivational variables that are expected to vary more rapidly (e.g., interest, strategies, goal orientation towards the task, attitudes towards the learning materials). This approach will provide very fine-grained data as to how motivation and affective states change based on changes in the learning environment or task structure (e.g., difficulty, problem type, topic, domain, etc.), as well as student interaction with the tutor or peers (e.g., strategies, cognitive processing, etc.).

Second, twice or three times a year we will administer questionnaires focused on constructs that may be semi-stable over time (e.g., self-efficacy, attitudes towards the domain, theory of intelligence, goal orientation towards the domain), a very traditional method in motivational research or research in SRL, although one whose shortcoming are increasingly being noticed (Zimmerman, 2008). Key to the current approach is that this more traditional type of data can be related to fine-grained data on PSLC measures of robust learning (instead of only using grades as a measure of learning which is typically the measure used in the literature in naturalistic learning settings). These questionnaires will be administered on paper, or perhaps electronically using SurveyMonkey or CTAT.

The data from these two approaches will enable the [[Metacognition_and_Motivation | Metacogniton and Motivation]] thrust to test path and structural equation models of the relation of particular M&M states and behaviors to robust learning (see Blackwell, Trzesniewski, and Dweck, 2007 for an example). Critically we will be linking motivational and affect variables to cognitive processes (by which they are hypothesized to do their work) and robust learning outcome measures. One goal of this work is theoretical integration of past work at the PSLC on instructional principles (macro-level), cognitive processes / knowledge components (mirco-level) and measures of robust learning to research / work and results on motivation and affect. Furthermore, this project provides a unique opportunity to test the generalizability of current and new theories of learning and motivation and affect across a number of academic domains (LearnLabs). This project will also play a critical role in the Theoretical Integration project of the thrust described in section 1.3.

These measures will be used in the experiments designed in the Social Communicative and Cognitive Factors Thrusts to provide across thrust integration. In addition, the collected data will be used to build and validate automated detectors for important aspects of students’ metacognition (described in the next section).

Our strategy will be initially to focus on a small set of variables that both builds on prior work conducted at the PSLC and the literature has identified as particularly relevant for learning in academic contexts.

*Awareness and use of SRL strategies (e.g., Motivated Strategies for Learning Questionnaire or its descendants) (Pintrich & de Groot, 1990)
*Self-efficacy (Bandura , 1997)
*Theory of intelligence (entity, incremental) (Dweck, 2006)
*Achievement goals (performance-approach, performance-avoidance, learning) (Darnon, Butera, & Harackiewicz, 2007; Elliot & Dweck, 1988)
*Interest (Hidi & Renninger, 2006)

==Results==
Forthcoming
==Explanation==
Forthcoming
===Future Plans===
===References===

Physics

2010-07-02T23:32:38Z

Bvds: /* Studies Conducted */ Partial Update to 2010, remove capacity table.

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time (2010), it is sited in:
* the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
* three courses at Watchung Hills Regional High School in Warren, NJ.
* one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

*Comparing two homework systems, Sophia Gershman, 2008-2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2009)

*An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System. Data mining study (Kasia Muldner, Winslow Burleson, Brett Van de Sande, Kurt VanLehn, 2009-2010).

In progress or planned:

*Comparing two homework systems, Sophia Gershman, 2010-2011.

*Reflective dialogs (Sandy Katz, Diane Litman, & Pam Jordan, 2010-2013).

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
* Michigan State University (1 instructor).
We see a shift in usage relative to previous years. Currently, the Naval Academy accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Fall 2008, between 284 and 617 different users (some use is anonymous, precluding an exact count) solved a total of 4853 Andes problems. The previous semesters, a total of 1647 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Gaming Behaviors in an Intelligent Tutoring System. To Appear in the International Conference on Intelligent Tutoring Systems, 2010, Pittsburgh, 10 pages.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System: Predictors and Impacts. Submitted to the
UMAISpecial Issue on Data Mining for Personalized Educational Systems.

==Current Status==

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally
teaches one section.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Analogical Scaffolding in Collaborative Learning

2010-07-02T23:23:41Z

Bvds: /* Connections */

== Analogical Scaffolding in Collaborative Learning ==
''Soniya Gadgil & Timothy Nokes''

=== Summary Table ===
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Soniya Gadgil (Pitt), Timothy Nokes (Pitt) <Br>
|-
| '''Other Contributers''' || Robert Shelby (USNA)
|-
| '''Study Start Date''' || Sept. 1, 2008
|-
| '''Study End Date''' || Aug. 31, 2009
|-
| '''LearnLab Site''' || United States Naval Academy (USNA)
|-
| '''LearnLab Course''' || Physics
|-
| '''Number of Students''' || ''N'' = 72
|-
| '''Total Participant Hours''' || 144 hrs.
|-
| '''DataShop''' || Anticipated
|}
<br>
=== Abstract ===
Past research has shown that collaboration can enhance learning in certain conditions. However, not much work has explored the cognitive mechanisms that underlie such learning. Chi, Hausmann and Roy (2004) propose three mechanisms including: self-explaining, other-directed explaining, and co-construction. In the current study, we will examine the use of these mechanisms when participants learn from [[worked examples]] across different collaborative contexts. We compare the effects of adding prompts that encourage [[analogical comparison]] to prompts that focus on single examples (non-comparison) to a traditional instruction condition, as students learn to solve Physics problems in the domain of rotational kinematics. Students learning processes will be analyzed by examining their verbal protocols. Learning will be assessed via [[robust learning|robust]] measures such as long-term retention and [[transfer]].

=== Background and Significance ===
Collaborative learning
Past research on collaborative learning provides compelling evidence that when students learn in groups of two or more, they show better learning gains at the group level than when working alone. Much of this research has focused on identifying conditions that underlie successful collaboration. For example, we know that factors such as presence of cognitive conflict (Schwartz, Neuman, & Biezuner, 2000), establishing of common ground (Clark, 2000) and scaffolding (or structuring) of the interaction are important factors affecting collaborative learning. Providing scripted problem solving activities (e.g., one participant plays the role of the tutor vs. tutee and then switch) have also been shown to facilitate collaborative learning compared to unscripted conditions (McLaren, Walker, Koedinger, Rummel, Spada, & Kalchman, 2007). These results are typically explained in terms of the sense making processes in which the structured collaborative environments provide the learner more opportunities to construct the relevant knowledge components.
Although much work has focused on improving learning through collaboration, little research has examined the cognitive processes underlying successful collaboration. Most of the prior work has focused on the outcome or product of the group and less has been concerned with the underlying processes that give rise to the product. If we can uncover the cognitive processes underlying collaborative learning, it can further our understanding of how to improve collaborative learning environments.

'''Schema Acquisition and Analogical Comparison''': A problem schema is a knowledge organization of the information associated with a particular problem category. Problem schemas typically include declarative knowledge of principles, concepts, and formulae, as well as the procedural knowledge for how to apply that knowledge to solve a problem. One way in which schemas can be acquired is through analogical comparison (Gick & Holyoak, 1983). Analogical comparison operates through aligning and mapping two example problem representations to one another and then extracting their commonalities (Gentner, 1983; Gick & Holyoak, 1983; Hummel & Holyoak, 2003). Research on analogy and schema learning has shown that the acquisition of schematic knowledge promotes flexible transfer to novel problems. For example, Gick and Holyoak (1983) found that transfer of a solution procedure was greater when participants’ schemas contained more relevant structural features. Analogical comparison has also been shown to improve learning even when both examples are not initially well understood (Kurtz, Miao, & Gentner, 2001; Gentner Lowenstein, & Thompson, 2003). By comparing the commonalities between two examples, students could focus on the causal structure and improve their learning about the concept. Kurtz et al. (2001) showed that students who were learning about the concept of heat transfer learned more when comparing examples than when studying each example separately.
In an ongoing project in the Physics LearnLab by Nokes & VanLehn, (2008) students learned to solve problems on rotational kinematics in one of the three conditions: read worked examples, self-explain worked examples, and engage in analogical comparison of worked examples. Preliminary results showed that the groups that self-explained and engaged in analogical comparison outperformed the read-only control on the far transfer tests. Our current project builds upon these results by applying them in a collaborative setting. In summary, prior work has shown that analogical comparison can facilitate schema abstraction and transfer of that knowledge to new problems. However, this work has not examined whether analogical scaffolding can lead to effective collaboration. The current work examines how analogical comparison may help students collaborate effectively.

=== Glossary ===
* [[collaboration]]
*[[analogical comparison]]
*[[in vivo experiment]]

=== Research Questions ===
* How can [[analogical comparison]] help students collaborate effectively?
* Can [[analogical comparison]] facilitate but also other learning mechanisms such as explanation, co-construction, and error-correction during collaboration?

=== Independent Variables ===
The only independent variable was Experimental Condition. There were three conditions: Compare, Non-compare, and Problem-solving.

*Compare Condition: Participants in this condition first read through and explained two worked examples. The worked examples did not have explanations for the solution steps and students were encouraged to generate the explanations and justifications for each step of the problem. They then performed the analogical comparison task, in which they were told that their task was to explicitly compare each part of the solution procedure to one another noting the similarities and differences between the two (e.g., goals, concepts, and solution procedures). Prompts in the form of questions to guide them through this process were provided. After a fixed amount of time, they were given the model answers to the questions and asked to check them against their own answers.

*Non-Compare Condition: Participants in this condition first read through a worked-out example. Similar to the non-compare condition, they were not given the explanations of the steps, and generated the explanations while working collaboratively. After reading through and explaining the first example they answered questions designed to act as prompts for the students to explain the worked example. These prompts were equivalent to the comparison prompts however they were only focused on a single problem (e.g., “what is the goal of this problem”). After a fixed amount of time, they were given the model answers to the questions and asked to check them against their own answers. They were then given a second worked example isomorphic to the first one. Again, students studied the example and generated explanations. They then answered questions based on the second worked example. After a fixed amount of time, they were provided answers to those questions.

*Problem-Solving Condition: The problem-solving condition served as a control condition and collaborated to solve problems without any scaffolding. Students in This condition received the same worked examples as the two experimental groups, but without any prompts to guide them through the problem-solving process. They were given additional problems for practice, to equate the time on task with the other two conditions.

=== Hypotheses ===
The following hypotheses are tested in the experiment:

1. Analogical scaffolding will serve as a script to enhance learning via collaboration, therefore students in the compare condition will outperform students in the other two conditions. Students in the compare and non-compare conditions will both outperform students in the control condition.

2. Students learning gains will differ by the kinds of learning processes they engaged in. Specifically, students engaging in self-explaining, other-directed explaining, and co-construction will show differential learning gains. This is an exploratory hypothesis and will be tested by undertaking a fine-grained analysis of verbal protocols generated by students as they solve problems collaboratively.

===Dependent Variables===
* [[Normal post-test]]: Near transfer, immediate: After training, students were given a post-test that assessed their learning on various measures. Specifically, 5 kinds of questions were included in the post-test.
* [[Robust learning]]
**Long-term retention: On the student’s regular mid-term exam, one problem was similar to the training. Since this exam occurred a week after the training, and the training took place in just under 2 hours, the student’s performance on this problem is considered a test of long-term retention.
**Near and far transfer: After training, students did their regular homework problems using Andes. Students did them whenever they wanted, but most completed them just before the exam.
**Accelerated future learning: The training was on rotational kinematics, and it was followed in the course by a unit on rotational dynamics. Log data from the rotational dynamics homework will be analyzed as a measure of acceleration of future learning.

=== Results ===

====Learning Results====
[[Image:Example.jpg|left]]During learning, students in all three conditions studied worked examples and solved isomorphic problems. The answers to compare questions and sequential questions were scored for students in those conditions. Students were given one point for every correct concept they mentioned while answering the analogical comparison questions in the compare condition or questions directed at studying individual examples in the sequential condition.
The sequential group answered a significantly higher percentage of questions (M = 70.28%, SE = 5%) correctly over the compare group (M = 50.66%, SE = 6%); F (1,22)= 5.60, p < 0.05.
Next, we looked at the isomorphic problems students solved during learning. On the first isomorphic problem, the three conditions were not significantly different; F (2,33) = 0.88, ns. On the second isomorphic problem, the three conditions were marginally different; F (2,33) = 2.60, p < 0.1 and the effect was in a direction favoring the sequential group over the compare and the problem solving groups.
High variation was observed in performance on the learning tasks, suggesting the possibility that individual differences would interact with learning outcomes. We are interested in testing the effectiveness of our intervention on test performance for when testing was successful. Therefore, we selected the best learners from each group by conducting a median split based on the learning scores (i.e., average scores on isomorphic problems from the learning phase). This was based on the assumption that there are some qualitative differences between learners who show high learning and those who show low learning during the learning intervention. This left us with six pairs in each group (high / low split for each condition).

====Test Results ====
The test phase was administered individually; therefore all scores reported below are means of scores for individual students.The test phase was divided into three sections: multiple-choice, problem solving, and open-ended questions.

* Multiple-Choice Test. Overall, all three conditions performed poorly on the multiple-choice questions. The overall mean was 3.82 (SE = 0.23) out of a total of ten points. There were no significant difference between conditions, F (2,69) = 0.05, ns. Therefore, we shall focus only on the performance of High learners. Item analysis of the multiple choice questions shows that the high learners in the compare condition performed significantly better than the high learners in the sequential and problem solving conditions on five questions. An ANOVA showed a significant difference between the three conditions, in a direction favoring the compare condition, F (2,33) = 3.86, p < 0.05 (See Fig. 1 for means and standard errors). Consistent with our predictions this result shows more conceptual learning for the compare condition than the sequential and problem solving conditions. Contrasts revealed that the compare group was significantly different from the sequential group t (1,33) = 2.56, p < 0.05 and problem-solving group; t (1,33) = 2.22, p < 0.05 but the sequential and problem-solving groups were not significantly different; t (1,33) = -0.34, ns.

* Problem Solving Test. The problem-solving test consisted of two questions, one of which was isomorphic to one of the problems the students had encountered in the learning intervention, but had different surface features. The other problem had extraneous values, which required students to determine which of the values were critical to solving the problem before they plugged in the numbers. We conducted a mixed model repeated measures ANOVA with problem-type as the within subject factor, and condition as the between-subject factor. There was a significant interaction problem-type X condition interaction (F (2,33)=3.37, p < 0.05). (See Fig. 2) Specifically, the students in the compare condition and sequential conditions performed better on the extraneous information problems than on the isomorphic problems, whereas students in the problem solving condition performed better on the isomorphic problem than they did on the extraneous information problem.
* Open-Ended Questions Test. The first question in this test consisted of two problems for which students had to determine whether the answer was correct or wrong and provide an explanation for the same. The second question consisted of two problems for which students had to calculate an answer and provide an explanation. Chi square tests revealed no difference between conditions on either question. All χ2s (2, N = 36) <4.8, ns.

===Further Information===
==== Annotated Bibliography ====
* Accepted as a poster presentation to CogSci 2009, Amsterdam.
* Presentation to the PSLC Industrial Affiliates, February, 2009
* Presentation to the PSLC Advisory Board, January, 2009
* Poster to be presented at the Second Annual Inter-Science of Learning Center Student and Post-Doc Conference (iSLC, '09) at Seattle, WA, February 2009

==== References ====
*Chase, W. G., & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 4, 55-81.
*Chen, Z. (1999). Schema induction in children’s analogical problem solving. Journal of Educational Psychology, 91, 703-715.
*Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.
*Chi, M. T. H., Roy, M., Hausmann, R. G. M. (2008). Observing Tutorial Dialogues Collaboratively: Insights About Human Tutoring Effectiveness From Vicarious Learning. Cognitive Science, 32, 301-341.
*Cooke, N., Salas, E., Cannon-Bowers, J.A., Stout, R. (2000). Measuring team knowledge. Human Factors 42, 151-173.
*Cummins, D. D. (1992). Role of analogical reasoning in the induction of problem categories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 1103-1124.
*Dunbar, K. (1999). The Scientist InVivo: How scientists think and reason in the laboratory. In Magnani, L., Nersessian, N., & Thagard, P. Model-based reasoning in scientific discovery. Plenum Press.
*Dunbar, K. (2001). The analogical paradox: Why analogy is so easy in naturalistic settings, yet so difficult in the psychology laboratory. In D. Gentner, Holyoak, K.J., & Kokinov, B. Analogy: Perspectives from Cognitive Science. MIT press.
*Gentner, D., Loewenstein, J., & Thompson, L. (2003). Learning and transfer: A general role for analogical encoding. Journal of Educational Psychology, 95, 393-408.
*Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15, 1-38.
* Hausmann, R. G. M., Chi, M. T. H. & Roy, M. (2004). Learning from collaborative problem solving: An analysis of three dialogue patterns. In the Twenty-sixth Cognitive Science Proceedings.
*Hummel, J. E., & Holyoak, K. J. (2003). A symbolic-connectionist theory of relational inference and generalization. Psychological Review, 110, 220-264.
*Kurtz, K. J., Miao, C. H., & Gentner, D. (2001). Learning by analogical bootstrapping. Journal of the Learning Sciences, 10, 417-446.
*Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208, 1335-1342.
*Lin, X. (2001). Designing metacognitive activities. Educational Technology Research & Development, 49, 1042-1629.
*McLaren, B., Walker, E., Koedinger, K., Rummel, N., & Spada, H. (2005). Improving algebra learning and collaboration through collaborative extensions to the algebra tutor. Conference on computer supported collaborative learning.
*Nokes, T. J. & VanLehn, K. (2008). Bridging principles and examples through analogy and explanation. In the Proceedings of the 8th International Conference of the Learning Sciences. Mahwah, Erlbaum.
*Novick, L. R., & Holyoak, K. J. (1991). Mathematical problem solving by analogy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 3, 398-415.
*Ohlsson, S. (1996). Learning from performance errors. Psychological Review, 103, 241-262.
*Paas, F. G. W. C., & Van Merrienboer, J. J. G. (1994). Variability of worked examples and transfer of geometrical problem solving skills: A cognitive-load approach. Journal of Educational Psychology, 86, 122-133.
*Palincsar, A. S., & Brown, A. L. (1984). Reciprocal Teaching of Comprehension-Fostering and Comprehension-Monitoring Activities. Cognition and Instruction, 1, 117-175.
*Schwarz, B. B., Neuman, Y., & Biezuner, S. (2000). Two Wrongs May Make a Right ... If They Argue Together! Cognition & Instruction, 18, 461-494.
*Ward, M., & Sweller, J. (1990). Structuring effective worked examples. Cognition and Instruction, 7, 1-39.

==== Connections ====
This project shares features with the following research projects:

* [[Bridging_Principles_and_Examples_through_Analogy_and_Explanation|Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn)]]
* [[Craig observing | Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi)]]

==== Future plans ====
Our future plans for January 2009 - August 2009:
* Code collaborative transcripts for different learning processes
* Conduct laboratory study

Physics

2010-07-02T23:20:23Z

Bvds: /* Publications on PLLC experiments */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time (2010), it is sited in:
* the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
* three courses at Watchung Hills Regional High School in Warren, NJ.
* one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 10 || 2 || 0 || 0 || 3 || 1 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn)]]

*Comparing two homework systems, Sophie Gershman, 2008--2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2008-2009)

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
* Michigan State University (1 instructor).
We see a shift in usage relative to previous years. Currently, the Naval Academy accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Fall 2008, between 284 and 617 different users (some use is anonymous, precluding an exact count) solved a total of 4853 Andes problems. The previous semesters, a total of 1647 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Gaming Behaviors in an Intelligent Tutoring System. To Appear in the International Conference on Intelligent Tutoring Systems, 2010, Pittsburgh, 10 pages.
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System: Predictors and Impacts. Submitted to the
UMAISpecial Issue on Data Mining for Personalized Educational Systems.

==Current Status==

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally
teaches one section.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2010-06-24T00:32:06Z

Bvds: /* Physics LearnLab Course */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time (2010), it is sited in:
* the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
* three courses at Watchung Hills Regional High School in Warren, NJ.
* one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 10 || 2 || 0 || 0 || 3 || 1 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn)]]

*Comparing two homework systems, Sophie Gershman, 2008--2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2008-2009)

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
* Michigan State University (1 instructor).
We see a shift in usage relative to previous years. Currently, the Naval Academy accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Fall 2008, between 284 and 617 different users (some use is anonymous, precluding an exact count) solved a total of 4853 Andes problems. The previous semesters, a total of 1647 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally
teaches one section.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2010-06-24T00:04:33Z

Bvds: /* Current Status */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ.

In order to increase the number of LearnLab sites, it is
essential that we increase the number of students using Andes.
During 2009, we plan to create a completely new web-based user interface. This will allow us to integrate Andes into [http://www.webassign.net WebAssign], the leading commercial provider of physics online homework making Andes easily accessible to over a hundred thousand students.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 10 || 2 || 0 || 0 || 3 || 1 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn)]]

*Comparing two homework systems, Sophie Gershman, 2008--2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2008-2009)

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
* Michigan State University (1 instructor).
We see a shift in usage relative to previous years. Currently, the Naval Academy accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Fall 2008, between 284 and 617 different users (some use is anonymous, precluding an exact count) solved a total of 4853 Andes problems. The previous semesters, a total of 1647 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally
teaches one section.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2009-03-25T07:05:22Z

Bvds: More details on Fall 2008 usage

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ.

In order to increase the number of LearnLab sites, it is
essential that we increase the number of students using Andes.
During 2009, we plan to create a completely new web-based user interface. This will allow us to integrate Andes into [http://www.webassign.net WebAssign], the leading commercial provider of physics online homework making Andes easily accessible to over a hundred thousand students.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 10 || 2 || 0 || 0 || 3 || 1 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn)]]

*Comparing two homework systems, Sophie Gershman, 2008--2009.

*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2008-2009)

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
* Michigan State University (1 instructor).
We see a shift in usage relative to previous years. Currently, the Naval Academy accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. During Fall 2008, between 284 and 617 different users (some use is anonymous, precluding an exact count) solved a total of 4853 Andes problems. The previous semesters, a total of 1647 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2009-01-21T20:55:56Z

Bvds: /* Studies Conducted */ missed study in table

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ.

In order to increase the number of LearnLab sites, it is
essential that we increase the number of students using Andes.
During 2009, we plan to create a completely new web-based user interface. This will allow us to integrate Andes into [http://www.webassign.net WebAssign], the leading commercial provider of physics online homework making Andes easily accessible to over a hundred thousand students.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 10 || 2 || 0 || 0 || 3 || 1 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*Nokes and Gadgil study at USNA, fall 2009.

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn)]]

*Comparing two homework systems, Sophie Gershman, 2008--2009.

*Nokes and Gadgil lab study.

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We see a shift in usage relative to previous years. Currently, the Naval Academy
accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2009-01-21T20:41:29Z

Bvds: /* Studies Conducted */ Add studies done, but not on Wiki pages.

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ.

In order to increase the number of LearnLab sites, it is
essential that we increase the number of students using Andes.
During 2009, we plan to create a completely new web-based user interface. This will allow us to integrate Andes into [http://www.webassign.net WebAssign], the leading commercial provider of physics online homework making Andes easily accessible to over a hundred thousand students.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 10 || 2 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*Nokes and Gadgil study at USNA, fall 2009.

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn)]]

*Comparing two homework systems, Sophie Gershman, 2008--2009.

*Nokes and Gadgil lab study.

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We see a shift in usage relative to previous years. Currently, the Naval Academy
accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2009-01-21T20:36:10Z

Bvds: /* Studies Conducted */ Move studies completed in 2008

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ.

In order to increase the number of LearnLab sites, it is
essential that we increase the number of students using Andes.
During 2009, we plan to create a completely new web-based user interface. This will allow us to integrate Andes into [http://www.webassign.net WebAssign], the leading commercial provider of physics online homework making Andes easily accessible to over a hundred thousand students.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We see a shift in usage relative to previous years. Currently, the Naval Academy
accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2009-01-21T18:43:47Z

Bvds: /* Physics LearnLab Course */ change sentence

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ.

In order to increase the number of LearnLab sites, it is
essential that we increase the number of students using Andes.
During 2009, we plan to create a completely new web-based user interface. This will allow us to integrate Andes into [http://www.webassign.net WebAssign], the leading commercial provider of physics online homework making Andes easily accessible to over a hundred thousand students.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We see a shift in usage relative to previous years. Currently, the Naval Academy
accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2009-01-21T18:38:22Z

Bvds: /* Advertising Andes in the physics community */ Add regional AAPT meeting.

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ.

In order to increase the number of LearnLab sites, it is
essential that we increase the number of students using Andes.
During 2009, we plan to create a completely new web-based user interface. This will allow us to integrate Andes into [http://www.webassign.net WebAssign], the leading commercial provider of physics online homework and make Andes readily accessible to over a hundred thousand students.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We see a shift in usage relative to previous years. Currently, the Naval Academy
accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008. Vendor exhibit.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
* Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.
In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2009-01-21T18:28:41Z

Bvds: fix sentence

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ.

In order to increase the number of LearnLab sites, it is
essential that we increase the number of students using Andes.
During 2009, we plan to create a completely new web-based user interface. This will allow us to integrate Andes into [http://www.webassign.net WebAssign], the leading commercial provider of physics online homework and make Andes readily accessible to over a hundred thousand students.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We see a shift in usage relative to previous years. Currently, the Naval Academy
accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-12-12T22:15:57Z

Bvds: /* Adoption of Andes */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We see a shift in usage relative to previous years. Currently, the Naval Academy
accounts for only 25% of our users; 32% of our users are now from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-12-12T22:15:17Z

Bvds: /* Adoption of Andes */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of Fall 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (1 instructor, several sections).
* SUNY Fredonia (1 instructor).
* Gannon University, Erie PA (1 instructor, several sections).
* Conant High School, Hoffman Estates, IL (1 instructor).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We see a shift in usage relative to previous years. Currently, the Naval Academy
accounts for only 25% of our users; 32% of our users are from High Schools.

We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Andes

2008-11-10T20:51:55Z

Bvds: change Andes website links

[[Category:Glossary]]

[http://www.andestutor.org Andes] is a physics tutoring system used in the PSLC [[Physics|Physics LearnLab]].

Physics

2008-11-10T20:47:16Z

Bvds: change Andes website links

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework. [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [[Andes]] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andestutor.org/AAPT-2007/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T16:15:33Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements requests that we hear regularly:
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).
** Instructor control over policy for student actions that are correct but don't contribute to a solution.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T16:12:53Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
* Other improvements that we hear regularly
** Allow sensitivity to lengths of vectors.
** Allow vector equations (currently, Andes equations are all scalar).

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T16:07:45Z

Bvds: /* Log file analysis */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:
* Download log files from OLI, anonymize them, and load them into the DataShop.
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T16:04:47Z

Bvds: /* Log file analysis */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Some activities associated with log files:
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T16:04:19Z

Bvds: /* Log file analysis */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present [[KC]]s vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Some activities associated with log files:
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T16:03:45Z

Bvds: /* Log file analysis */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Some activities associated with log files:
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T16:03:17Z

Bvds: /* Log file analysis */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [knowledge component]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Some activities associated with log files:
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T16:00:54Z

Bvds: /* Plans */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

===Log file analysis===
The [knowledge components] (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Some activities associated with log files:
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
* Finish investigating whether time spent can be a useful metric of student learning.
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.

Physics

2008-04-29T15:52:44Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:50:48Z

Bvds: /* Supporting existing Andes users */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:49:44Z

Bvds: /* Grading policy */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:46:53Z

Bvds: /* Grading policy= */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools.

===Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:46:12Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools.

==Grading policy===
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
There are two problems:
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:39:49Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:35:16Z

Bvds: /* Increase awareness of Andes */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:34:49Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:31:06Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we plan to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:30:19Z

Bvds: /* Supporting existing Andes users */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we envision to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add instructor requested problem types (such as graph drawing).
* Fixing instructor reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Physics

2008-04-29T15:26:46Z

Bvds: /* Supporting existing Andes instructors */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we envision to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes users===
There are a number of non-PLLC instructors using Andes in their classrooms. We need to be responsive to instructor feedback:
* Add requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add requested problem types (such as graph drawing).
* Fixing instructor-reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other instructors.

Physics

2008-04-29T15:26:21Z

Bvds: /* Supporting existing Andes instructors */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we envision to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes instructors===
There are a number of non-PLLC instructors using Andes in their classrooms. We need to be responsive to instructor feedback:
* Add requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add requested problem types (such as graph drawing).
* Fixing instructor-reported bugs promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other instructors.

Physics

2008-04-29T15:19:33Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA]. Furthermore, we envision to offer Andes under an Open Source License, to ensure long-term availability and allow others to contribute to the future development of Andes.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes instructors===
There are a number of non-PLLC instructors using Andes in their classrooms. We need to be responsive to instructor feedback:
* Add requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add requested problem types (such as graph drawing).
* Fixing instructor-reported bugs promptly.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other instructors.

Physics

2008-04-29T15:17:27Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our partners, [http://www.webassign.net WebAssign] and [http://www.lon-capa.org LON-CAPA].
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes instructors===
There are a number of non-PLLC instructors using Andes in their classrooms. We need to be responsive to instructor feedback:
* Add requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add requested problem types (such as graph drawing).
* Fixing instructor-reported bugs promptly.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other instructors.

Physics

2008-04-29T15:15:29Z

Bvds: /* Supporting existing Andes instructors */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our commercial partner, WebAssign.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes instructors===
There are a number of non-PLLC instructors using Andes in their classrooms. We need to be responsive to instructor feedback:
* Add requested homework problems. We will continue our policy of adding new content based on instructor requests.
* Add requested problem types (such as graph drawing).
* Fixing instructor-reported bugs promptly.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other instructors.

Physics

2008-04-29T15:14:38Z

Bvds: /* Improvements to Andes itself */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our commercial partner, WebAssign.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes instructors===
There are a number of non-PLLC instructors using Andes in their classrooms. We need to be responsive to instructor feedback:
* Add requested homework problems,
* Add requested problem types (such as graph drawing).
* Fixing instructor-reported bugs promptly.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other instructors.

Physics

2008-04-29T15:12:32Z

Bvds: /* Plans */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses, and make their experience a positive one.

===Increase awareness of Andes===
We need to increase awareness of Andes in the physics community.
To date, we have focused our efforts on national meetings of
the AAPT and APS. However, we plan to broaden our efforts:
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
* Continue to visiting physics departments at other universities.
* Publish PLLC-related research in the physics education journals.

===Web-based delivery===
Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.

===Improvements to Andes itself===
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
* Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
* Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our commercial partner, WebAssign.
* Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
* Instructors want good, effective hints.
* Instructors want Andes to cover an entire course.
* Instructors want a grading policy that is understandable and customizable.

===Supporting existing Andes instructors===
There are a number of non-PLLC instructors using Andes in their classrooms. We need to be responsive to instructor feedback:
* Add requested homework problems,
* Add requested problem types (such as graph drawing).
* Fixing instructor-reported bugs promptly.
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other instructors.

Physics

2008-04-29T14:48:57Z

Bvds: /* Publications on PLLC experiments */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses. There are three strategies that we are pursuing.
*First, we need to increase awareness of Andes in the general physics community. We have given talks, presented posters, and given a workshop at the American Associate of Physics Teachers (AAPT) conferences and other national meetings attended by physics instructors. We have begun to promote Andes at regional AAPT meetings and hope to continue this in the future. Finally, we plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
*Second, Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.
* Third, based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
** Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
** Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our commercial partner, WebAssign.
** Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
** Instructors want good, effective hints. Instructors want Andes to cover an entire course.
** Instructors want a grading policy that is understandable and customizable.

* '''Increasing Andes awareness in the physics community'''
**Present talks and posters at the American Association of Physics Teachers conference in January 2008 and the American Physics Society (APS) meeting in April 2008.
**Continue to visiting physics departments at other universities.
** Publish Andes-based research in the physics education journals.
* '''Supporting existing Andes instructors''': There are a number of non-PLLC instructors using Andes in their classrooms. We need to make their experience with Andes a positive one. Hopefully, a positive experience will lead to interest in participating in LearnLab. This includes:
**Adding instructor-requested homework problems
**Fixing instructor-reported bugs promptly, and
**Including some instructor control over the hinting behavior of Andes.

* '''Increasing instructor acceptance''': In order to increase the number of instructors using Andes, we need to address the following issues:
** '''Improving hints given to students''': [[Andes]] is supposed to mimic the hints that an expert (human) tutor would give to students. However, sometimes the hints are misleading or do not help the student better understand the problem at a deep level.
** '''Non-obvious conventions''': Communicating physics and math precisely and unambigously entails using some notational and user interface conventions that user find non-obvious. We call these non-obvious conventions (NOCs; pronounced “knocks”). We need to collect as many non-obvious conventions as we can think of and fix them or highlight them in the videos and other training.
** '''Lesser priority items''': There are a number of improvements to Andes that would lead to increased instructor happiness.
*** '''Vectors in equations''': Handling of equivalent variables. And the equations that use them.
*** '''Scale drawing of vectors.'''
*** '''True but irrelevant entries''': Currently these turn red. Instructor should be able to select the color and/or the warning that appears. They should be able to have separate policies for equations and non-equations.

Physics

2008-04-29T14:46:36Z

Bvds: /* Publications on PLLC experiments */

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007.''
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses. There are three strategies that we are pursuing.
*First, we need to increase awareness of Andes in the general physics community. We have given talks, presented posters, and given a workshop at the American Associate of Physics Teachers (AAPT) conferences and other national meetings attended by physics instructors. We have begun to promote Andes at regional AAPT meetings and hope to continue this in the future. Finally, we plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
*Second, Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.
* Third, based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
** Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
** Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our commercial partner, WebAssign.
** Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
** Instructors want good, effective hints. Instructors want Andes to cover an entire course.
** Instructors want a grading policy that is understandable and customizable.

* '''Increasing Andes awareness in the physics community'''
**Present talks and posters at the American Association of Physics Teachers conference in January 2008 and the American Physics Society (APS) meeting in April 2008.
**Continue to visiting physics departments at other universities.
** Publish Andes-based research in the physics education journals.
* '''Supporting existing Andes instructors''': There are a number of non-PLLC instructors using Andes in their classrooms. We need to make their experience with Andes a positive one. Hopefully, a positive experience will lead to interest in participating in LearnLab. This includes:
**Adding instructor-requested homework problems
**Fixing instructor-reported bugs promptly, and
**Including some instructor control over the hinting behavior of Andes.

* '''Increasing instructor acceptance''': In order to increase the number of instructors using Andes, we need to address the following issues:
** '''Improving hints given to students''': [[Andes]] is supposed to mimic the hints that an expert (human) tutor would give to students. However, sometimes the hints are misleading or do not help the student better understand the problem at a deep level.
** '''Non-obvious conventions''': Communicating physics and math precisely and unambigously entails using some notational and user interface conventions that user find non-obvious. We call these non-obvious conventions (NOCs; pronounced “knocks”). We need to collect as many non-obvious conventions as we can think of and fix them or highlight them in the videos and other training.
** '''Lesser priority items''': There are a number of improvements to Andes that would lead to increased instructor happiness.
*** '''Vectors in equations''': Handling of equivalent variables. And the equations that use them.
*** '''Scale drawing of vectors.'''
*** '''True but irrelevant entries''': Currently these turn red. Instructor should be able to select the color and/or the warning that appears. They should be able to have separate policies for equations and non-equations.

Physics

2008-04-29T14:46:05Z

Bvds: /* Publications on PLLC experiments */ add 2007

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007.''
Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses. There are three strategies that we are pursuing.
*First, we need to increase awareness of Andes in the general physics community. We have given talks, presented posters, and given a workshop at the American Associate of Physics Teachers (AAPT) conferences and other national meetings attended by physics instructors. We have begun to promote Andes at regional AAPT meetings and hope to continue this in the future. Finally, we plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
*Second, Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.
* Third, based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
** Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
** Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our commercial partner, WebAssign.
** Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
** Instructors want good, effective hints. Instructors want Andes to cover an entire course.
** Instructors want a grading policy that is understandable and customizable.

* '''Increasing Andes awareness in the physics community'''
**Present talks and posters at the American Association of Physics Teachers conference in January 2008 and the American Physics Society (APS) meeting in April 2008.
**Continue to visiting physics departments at other universities.
** Publish Andes-based research in the physics education journals.
* '''Supporting existing Andes instructors''': There are a number of non-PLLC instructors using Andes in their classrooms. We need to make their experience with Andes a positive one. Hopefully, a positive experience will lead to interest in participating in LearnLab. This includes:
**Adding instructor-requested homework problems
**Fixing instructor-reported bugs promptly, and
**Including some instructor control over the hinting behavior of Andes.

* '''Increasing instructor acceptance''': In order to increase the number of instructors using Andes, we need to address the following issues:
** '''Improving hints given to students''': [[Andes]] is supposed to mimic the hints that an expert (human) tutor would give to students. However, sometimes the hints are misleading or do not help the student better understand the problem at a deep level.
** '''Non-obvious conventions''': Communicating physics and math precisely and unambigously entails using some notational and user interface conventions that user find non-obvious. We call these non-obvious conventions (NOCs; pronounced “knocks”). We need to collect as many non-obvious conventions as we can think of and fix them or highlight them in the videos and other training.
** '''Lesser priority items''': There are a number of improvements to Andes that would lead to increased instructor happiness.
*** '''Vectors in equations''': Handling of equivalent variables. And the equations that use them.
*** '''Scale drawing of vectors.'''
*** '''True but irrelevant entries''': Currently these turn red. Instructor should be able to select the color and/or the warning that appears. They should be able to have separate policies for equations and non-equations.

Physics

2008-04-29T14:20:33Z

Bvds: /* Achievements */ typo

= Physics LearnLab Course =

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time, it is sited in the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD and three courses at Watchung Hills Regional High School in Warren, NJ. We are actively seeking additional sites both at the high school and university level.

Students in PLLC classes use the [http://www.andes.pitt.edu Andes] intelligent tutoring system to do their homework. [http://www.andes.pitt.edu Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

== Studies Conducted ==

{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
|+ '''Summary of Studies'''
|-
!
! colspan=2 | ''In Vivo''
! colspan=2 | Pull Out
! colspan=2 | Lab
! colspan=4 | Capacity
|-
! Course || Run || Planned || Run || Planned || Run || Planned
! Total # Sections
! Total # Students
! Max # Studies / Year
! Max # Students / Study
|-
| Physics || 9 || 1 || 0 || 0 || 3 || 0 || 5 || 130 || 4 || 65
|}

Capacity was determined by counting the number of students who solved more than 40 [http://www.andes.pitt.edu Andes] problems in Fall 2007. There are about 25 students in a section and each LearnLab site has about 65 students.

Completed studies:

*[[Ringenberg_Examples-as-Help | Scaffolding Problem Solving with Embedded Example to Promote Deep Learning (Ringenberg & VanLehn, 2005)]]

*[[Hausmann_Diss|The effects of elaborative dialog on problem solving and learning (Hausmann & Chi, 2005)]]

*[[Post-practice reflection (Katz)|Post-practice reflection (Katz & Connelly, 2005)]]

*[[Craig_questions|Deep-level questions during example studying (Craig & Chi, 2006)]]

*[[Hausmann_Study|Does it matter who generates the explanations? (Hausmann & VanLehn, 2006)]]

*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]

*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]

*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]

*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]

*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]

In progress or planned:

*[[Ringenberg Ill-Defined Physics|Does Solving Ill-Defined Physics Problems Elicit More Learning than Conventional Problem Solving? (Ringenberg & VanLehn, 2008)]]

*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]

*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]

==Achievements==

From its inception in January 2005 to the present, we have achieved the following:

===Content development milestones===
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
* We have increased the total number of working Andes problems from 350 to 556.
* The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

===Enabling Technologies===
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI]. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007). The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].

===Log file analysis===
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
* Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
* Conducted a [http://www.andes.pitt.edu/workshop/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007]. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

===Adoption of Andes===
As of spring 2008, [[Andes]] is being used at the following institutions:
* St. Anselm college, Manchester NH (1 instructor).
* US Naval Academy (2 instructors, several sections).
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course]. From January to April 2008, between 90 and 278 different users (some use is anonymous, precluding an exact count) solved a total of 1647 Andes problems. The previous semester, a total of 1260 problems were solved.

===Advertising Andes in the physics community===
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn. Changing Student Attitudes using Andes, An Intelligent Homework System. Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An intelligent homework helper]. Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Changing Student Attitudes using Andes, An Intelligent Homework System.] Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
* B. van de Sande, & R. Hausman. An Analysis of Student Learning Using the Andes Homework System. Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill. Andes: An Intelligent Tutor Homework System. Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab. Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007. [http://www.andes.pitt.edu/workshop/ Workshop website].
* S. Katz & J. Connelly. Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes. Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill. Student attitudes towards Andes, an intelligent tutor homework system. Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration?] Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],'' Youngstown OH, March 2008.
* B. van de Sande, & R. Hausmann. Does an intelligent tutor homework system encourage beneficial collaboration? Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings. We manned a [http://www.learnlab.org PSLC] vendor exhibit at:
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)], Youngstown OH, March 2008.
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.
In addition, we have presented Andes at other places: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

===Publications on Andes===
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). ''Proceedings of User Modelling 2007.''
* VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.''

===Publications on PLLC experiments===
* Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007.''
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
* Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 548-550. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 551-553. Amsterdam, Netherlands: IOS Press.
* Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 409-416. Amsterdam, Netherlands: IOS Press.
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 554-556. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.
* Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.

==Current Status==

The PLLC at the US Naval Academy is currently comprised of 3-5 sections (depending on the semester) of 25 students each. The sections are taught by Professors Mary Wintersgill and Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [http://www.andes.pitt.edu Andes], and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

==Plans==

Our major goal continues to be to expand the number of sites and instructors involved in the PLLC. There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters. In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses. There are three strategies that we are pursuing.
*First, we need to increase awareness of Andes in the general physics community. We have given talks, presented posters, and given a workshop at the American Associate of Physics Teachers (AAPT) conferences and other national meetings attended by physics instructors. We have begun to promote Andes at regional AAPT meetings and hope to continue this in the future. Finally, we plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
*Second, Andes currently runs on Microsoft Windows machines as a Windows executable, requiring a software download/installation before it can be run. We lost at least two potential sites (Paul Perkins’ High School class in Bellevue WA and the US Air Force Academy) due to issues associated with this. In both cases, instructors were enthusiastic about Andes and assigned Andes to their students, but a significant number of students had troubles installing the software, getting it to run reliably, or did not have Microsoft Windows available to them. We believe we are losing many other potential clients due to this architecture. Thus, we have begun the development of a new web-based user interface to allow delivery of Andes as a true web application.
* Third, based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
** Instructors want a user interface that appears to be simple to learn. The new version of the user interface that we are developing will have a very simple design, making it similar to a generic drawing program (like Powerpoint).
** Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. The new user web-based interface will allow us to deliver Andes via our commercial partner, WebAssign.
** Instructors want all reasonable student actions to be accepted. The new user interface will feature free text input, allowing greater flexibility.
** Instructors want good, effective hints. Instructors want Andes to cover an entire course.
** Instructors want a grading policy that is understandable and customizable.

* '''Increasing Andes awareness in the physics community'''
**Present talks and posters at the American Association of Physics Teachers conference in January 2008 and the American Physics Society (APS) meeting in April 2008.
**Continue to visiting physics departments at other universities.
** Publish Andes-based research in the physics education journals.
* '''Supporting existing Andes instructors''': There are a number of non-PLLC instructors using Andes in their classrooms. We need to make their experience with Andes a positive one. Hopefully, a positive experience will lead to interest in participating in LearnLab. This includes:
**Adding instructor-requested homework problems
**Fixing instructor-reported bugs promptly, and
**Including some instructor control over the hinting behavior of Andes.

* '''Increasing instructor acceptance''': In order to increase the number of instructors using Andes, we need to address the following issues:
** '''Improving hints given to students''': [[Andes]] is supposed to mimic the hints that an expert (human) tutor would give to students. However, sometimes the hints are misleading or do not help the student better understand the problem at a deep level.
** '''Non-obvious conventions''': Communicating physics and math precisely and unambigously entails using some notational and user interface conventions that user find non-obvious. We call these non-obvious conventions (NOCs; pronounced “knocks”). We need to collect as many non-obvious conventions as we can think of and fix them or highlight them in the videos and other training.
** '''Lesser priority items''': There are a number of improvements to Andes that would lead to increased instructor happiness.
*** '''Vectors in equations''': Handling of equivalent variables. And the equations that use them.
*** '''Scale drawing of vectors.'''
*** '''True but irrelevant entries''': Currently these turn red. Instructor should be able to select the color and/or the warning that appears. They should be able to have separate policies for equations and non-equations.