Difference between revisions of "Physics"

From LearnLab
Jump to: navigation, search
(→‎Achievements: add Ericsson paper)
(→‎Plans: Update to match new version of Andes)
 
(95 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
= Physics LearnLab Course =
 
= Physics LearnLab Course =
  
−
The Physics LearnLab Course (PLLC) provides a research facility for studying how students learn physics.  It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction.  Students in PLLC classes use the [[Andes]] intelligent homework helper to do their homework.  [[Andes]] not only helps students learn physics, as demonstrated in earlier evaluations, but it also allows the PLLC to collect fine-grained data on student activity while solving homework problems during the entire course.  Other elements of the course are taught in a traditional manner with lectures, labs, and a commercial paper-based textbook.   In-vivo experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.  
+
The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics.  It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction.  At this time (2010), it is sited in:
 +
* the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
 +
* three courses at Watchung Hills Regional High School in Warren, NJ.
 +
* one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.
  
−
Currently, the Physics LearnLab Course is sited in the two-semester Introductory Physics course at the US Naval Academy in Annapolis, MD and in three courses at Watchung Hills Regional High School in Warren, NJ.  At each institution, it consists of 3-5 sections (depending on the experiment) of 25 students each.  A major goal is to increase the number courses using [[Andes]] and, from that, to increase the number of sites participating in LearnLab studies.  
+
 
 +
Students in PLLC classes use the [http://www.andestutor.org Andes] intelligent tutoring system to do their homework.  [http://www.andestutor.org Andes] allows the PLLC to collect fine-grained data on student activity through the entire semester.  The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook.  ''In vivo'' experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.
  
 
== Studies Conducted ==
 
== Studies Conducted ==
−
 
−
{| border="1" cellspacing="0" cellpadding="5" style="text-align: center;"
 
−
|+ '''Summary of Studies'''
 
−
|-
 
−
!
 
−
! colspan=2 | In Vivo
 
−
! colspan=2 | Pull Out
 
−
! colspan=2 | Lab
 
−
! colspan=4 | Capacity
 
−
|-
 
−
! Course || Run || Planned || Run || Planned || Run || Planned
 
−
! Total # Sections
 
−
! Total # Students
 
−
! Max # Studies / Year
 
−
! Max # Students / Study
 
−
|-
 
−
| Physics || 9 || 1 || 0 || 0 || 2 || 0 || 5 || 120 || 4 || 120
 
−
|}
 
  
 
Completed studies:
 
Completed studies:
Line 39: Line 25:
 
*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]
 
*[[Craig_observing|Learning from Problem Solving while Observing Worked Examples (Craig Gadgil, & Chi, 2007)]]
  
−
*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2007)]]
+
*[[Reflective Dialogues (Katz)|Reflective Dialogues (Katz, Connelly & Treacy, 2006-2007)]]
  
 
*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]
 
*[[Hausmann_Study2|The effects of interaction on robust learning (Hausmann & VanLehn, 2007)]]
Line 45: Line 31:
 
*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]
 
*[[Bridging_Principles_and_Examples_through_Analogy_and_Explanation | Bridging Principles and Examples through Analogy and Explanation (Nokes & VanLehn, 2007)]]
  
−
Planned:
+
*[[Extending Reflective Dialogue Support (Katz & Connelly)|Extending Automated Dialogue Support for Robust Learning of Physics (Katz & Connelly, 2007-2008)]]
 +
 
 +
*[[Plateau_study|The Interaction Plateau: A comparison between human tutoring, Andes, and computer-aided instruction (Hausmann, van de Sande, & VanLehn, 2008)]]
 +
 
 +
*[[Self-explanation: Meta-cognitive vs. justification prompts|Self-explanation: Meta-cognitive vs. justification prompts (Hausmann, van de Sande, Gershman, & VanLehn, 2008)]]
 +
 
 +
*[[Analogical Scaffolding in Collaborative Learning|Analogical Scaffolding in Collaborative Learning (Gadgil &Nokes, 2008-2009)]]
 +
 
 +
*Comparing two homework systems, Sophia Gershman, 2008-2009.
 +
 
 +
*Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2009)
 +
 
 +
*An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System.  Data mining study (Kasia Muldner, Winslow Burleson, Brett Van de Sande, Kurt VanLehn, 2009-2010).
 +
 
 +
In progress or planned:
  
−
*Hausmann study at Watchung Hills RHS [needs page].
+
*Comparing two homework systems, Sophia Gershman, 2010-2011.
  
−
*Hausman & VanLehn lab study at Pitt [needs page].
+
*Reflective dialogs (Sandy Katz, Diane Litman, & Pam Jordan, 2010-2013).
  
 
==Achievements==
 
==Achievements==
Line 55: Line 55:
 
From its inception in January 2005 to the present, we have achieved the following:  
 
From its inception in January 2005 to the present, we have achieved the following:  
  
−
*  '''Content development milestones'''
+
===Content development milestones===
−
** The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.  
+
* The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.  
−
** We have increased the total number of  working Andes problems from 350 to 556.
+
* We have increased the total number of  working Andes problems from 350 to 556.
−
** The number of physics principles has increased from 126 to 219.  The number of  rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915.  The number of scalar quantities defined in Andes has increased from 85 to 126.   
+
* The number of physics principles has increased from 126 to 219.  The number of  rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915.  The number of scalar quantities defined in Andes has increased from 85 to 126.   
−
** We shot videos of problems being solved—at least one per problem set—and revised many of the older videos.  These act as worked examples.  Students who view the videos in a problem set before solving any problems have a much easier time of it.  
+
* We shot videos of problems being solved—at least one per problem set—and revised many of the older videos.  These act as worked examples.  Students who view the videos in a problem set before solving any problems have a much easier time of it.
 +
 
 +
===Enabling Technologies===
 +
* We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI].  In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
 +
* We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
 +
* Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order.  This was needed for the Sandy Katz experiment in fall 2006.
 +
* Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007).  The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].
  
−
*  '''Enabling Technologies'''
+
===Log file analysis===
−
**We developed a way to run Andes under [http://www.cmu.edu/oli/ OLI].  In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.  
+
The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments.  We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.
−
**We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
+
* Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?).  Investigated whether time-on-task could be used as a metric for student learning of KC's.  
−
**Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order.  This was needed for the Sandy Katz experiment in fall 2006.
+
* Begun comparing the Log data to end-of-semester surveys administered at the USNA.  The surveys were not anonymous, so individual survey results can be matched with the associated log files.
−
**Andes raw logs can now be converted to the [[http://learnlab.web.cmu.edu/ DataShop]] format at the [[knowledge component]] level (June 2007).  The knowledge components associated with each correct student action (corresponding with a [[step]]) and must incorrect action (see [[transaction]]) is determined by [[Andes]].
+
* Conducted a [http://www.andestutor.org/AAPT-2007/ workshop on log file analysis] at [http://web.phys.ksu.edu/perc2007/ PERC 2007].  Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.
  
−
*  '''Advertising Andes in the physics community'''
+
===Adoption of Andes===
−
The following lists activities by Brett van de Sande.
+
As of Spring 2010, [[Andes]] is being used at the following institutions:
−
**Presentations at the [http://www.aapt.org American Association of Physics Teachers (AAPT)] conferences:
+
* St. Anselm college, Manchester NH (1 instructor).
−
***[http://www.aapt.org/Events/sm2006 Syracuse NY (2006)] 1 talk, 2 posters;
+
* US Naval Academy (1 instructor, several sections).
−
***[http://www.aapt.org/Events/wm2007 Seattle WA (2007)] 1 talk, 1 poster;
+
* Upper St. Clair School District, PA (1 instructor).
−
***[http://www.aapt.org/Events/sm2007 Greensboro NC (2007)] 1 talk, 1 workshop, 3 posters;
+
* Clark County School District, NV (1 instructor, distance learning).
−
***[http://www.aapt.org/Events/wm2008 Baltimore MD (2008)] 1 talk, 1 poster.
+
* Clark College, WA (1 instructor, several sections)
−
**Presentations at AAPT regional meetings
+
* Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).
−
***Western PA/Ohio Regional meeting March 2008, Youngstown OH.
 
−
***Central PA Regional meeting April 2008, Lock Haven PA.
 
−
**Presentations at the [http://www.aps.org/ American Physical Society] Meeting in Denver, CO, March 2007.
 
−
**Presented talks on Andes at Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).
 
  
−
*  '''Pulblications on Andes''':
+
We observe steadily growing use of Andes by individuals not enrolled in any [http://www.cmu.edu/oli/courses/physics/ OLI course].  During Spring 2010, between 285 and 1332 different users (some use is anonymous, precluding an exact count) solved a total of 15192 Andes problems.  The previous semester, a total of 11819 problems were solved.
−
**VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M.  The Andes Physics Tutoring System: Lessons Learned.  ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
 
−
**Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C.  The Andes physics tutoring system: Five years of evaluations.  In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.''  (pp. 678-685) Amsterdam, Netherlands: IOS Press.
 
−
**Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A.  (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
 
−
**VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step?  Toward general, abstract representations of tutoring system log data.  In C. Conati & K. McCoy (eds).  ''Proceedings of User Modelling 2007.''
 
−
**VanLehn, K., & van de Sande, B.  (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise:  Toward measurement of expert performance and design of optimal learning environments.''
 
  
−
*  '''Publications on PLLC experiments''':
+
===Advertising Andes in the physics community===
−
**Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press.
+
We have focused our efforts on meetings of the [http://www.aapt.org American Association of Physics Teachers (AAPT)] and the [http://www.aps.org/meetings/ American Physical Society (APS)] where we have presented numerous talks, posters, and a workshop. 
−
**Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007.''
+
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill.    Andes: An Intelligent Tutor for Introductory Physics Homework. Contributed talk at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
−
**Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
+
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill.    Andes: An Intelligent Tutor Homework System for Introductory Physics. Poster at the ''[http://www.aapt.org/Events/sm2006 2006 AAPT Summer Meeting],'' Syracuse NY, July 2006.
−
**Chi, Min & VanLehn, K.  (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.''  pp. 548-550. Amsterdam, Netherlands: IOS Press.  
+
* B. van de Sande, R. Hausman, R. Shelby, D. Treacy, & K. VanLehn. Andes: An Intelligent Homework System for Introductory Physics. Contributed talk at the [http://www.aapt.org/Events/wm2007 ''2007 AAPT Winter Meeting''], Seattle WA, January 2007.
−
**Chi, Min & VanLehn, K.  (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.'' pp. 551-553.  Amsterdam, Netherlands: IOS Press.
+
* B. van de Sande, R. Shelby, D. Treacy, & K. VanLehn.  Changing Student Attitudes using Andes, An Intelligent Homework System.  Poster at the ''[http://www.aapt.org/Events/wm2007 2007 AAPT Winter Meeting],'' Seattle WA, January 2007.
−
**Chi, Min & VanLehn, K.  (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.''  pp. 409-416.  Amsterdam, Netherlands: IOS Press.
+
* [http://meetings.aps.org/link/BAPS.2007.MAR.A21.10 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill.  Andes: An intelligent homework helper].  Contributed talk at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
−
**Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.''  pp. 554-556. Amsterdam, Netherlands: IOS Press.  
+
* [http://meetings.aps.org/link/BAPS.2007.MAR.K1.199 B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill.  Changing Student Attitudes using Andes, An Intelligent Homework System.]  Poster at the [http://meetings.aps.org/Meeting/MAR07 ''2007 APS March Meeting''], Denver CO, March 2007.
−
**Hausmann, R. G. M. & VanLehn, K. (2007).  Explaining self-explaining:  A contrast between content and generation.  In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.  
+
* B. van de Sande, & R. Hausman.  An Analysis of Student Learning Using the Andes Homework System.  Contributed talk at the [http://www.aapt.org/Events/sm2007 ''2007 AAPT Summer Meeting''], Greensboro NC, July 2007.
−
**Hausmann, R. G. M. & VanLehn, K. (2007).  Self-explaining in the classroom:  Learning curve evidence  In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
+
* B. van de Sande, R. Shelby, D. Treacy, K. VanLehn, & M. Wintersgill.  Andes: An Intelligent Tutor Homework System.  Poster at the ''[http://www.aapt.org/Events/sm2007 2007 AAPT Summer Meeting],'' Greensboro NC, July 2007.
−
** Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
+
* B. van de Sande & K. VanLehn. Cognitive Analysis of Student Learning Using LearnLab.  Workshop presented at the [http://web.phys.ksu.edu/perc2007/ ''Physics Education Research Conference''], Greensboro NC, August 2007.  [http://www.andestutor.org/AAPT-2007/ Workshop website].
−
** Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
+
* S. Katz & J. Connelly.  Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes.  Poster presented at the ''[http://web.phys.ksu.edu/perc2007/ Physics Education Research Conference],'' Greensboro NC, August 2007.
−
** Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
+
* B. van de Sande, & R. Hausmann.  Does an intelligent tutor homework system encourage beneficial collaboration?  Contributed talk at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],''  Baltimore MD, January 2008.
 +
* B. van de Sande, R. Shelby, D. Treacy, & M. Wintersgill.  Student attitudes towards Andes, an intelligent tutor homework system.  Poster at the ''[http://www.aapt.org/Events/wm2008 2008 AAPT Winter Meeting],'' Baltimore MD, January 2008.
 +
* [http://meetings.aps.org/Meeting/OSS08/Event/86059 B. van de Sande, & R. Hausmann.  Does an intelligent tutor homework system encourage beneficial collaboration?]  Contributed talk at ''[http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],''  Youngstown OH, March 2008.
 +
* B. van de Sande, & R. Hausmann.  Does an intelligent tutor homework system encourage beneficial collaboration?  Contributed talk at the ''Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT),'' Lock Haven PA, April 2008.
 +
* K. Muldner, W. Burleson, B. van de Sande, & K. VanLehn, Fun and Gaming with Andes.  Poster presented at the [http://www.aapt.org/Conferences/sm2010/ 2010 AAPT Summer meeting], Portland Oregon, July 2010.
 +
* B. van de Sande. Experimental methods for studying student Metacognition and Affect.  Workshop presented at the [http://www.compadre.org/per/conferences/2010/ ''Physics Education Research Conference 2010''], Portland OR, July 2010.
 +
These meetings generally do not publish proceedings.
 +
 +
More recently, we have begun promoting the Physics LearnLab at regional [http://www.aapt.org AAPT] meetings:
 +
* [http://www.ysu.edu/osaps2008/ Touring the Electromagnetic Spectrum (OSAPS 2008)],  Youngstown OH, March 2008.  Vendor exhibit.
 +
* Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008.  Vendor exhibit.
 +
*  Fall meeting of the Arizona section of the AAPT, October 2008.  Workshop for instructors.
 +
In addition, we have presented Andes at other universities:  Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).
 +
 
 +
===Publications on Andes===
 +
* VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M.  The Andes Physics Tutoring System: Lessons Learned.  ''International Journal of Artificial Intelligence and Education,'' 15 (3), 1-47.
 +
* Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C.  The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), ''Artificial Intelligence in Education.'' (pp. 678-685) Amsterdam, Netherlands: IOS Press.
 +
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
 +
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step?  Toward general, abstract representations of tutoring system log data.  In C. Conati & K. McCoy (eds).  ''Proceedings of User Modelling 2007.''  
 +
* VanLehn, K., & van de Sande, B.  (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), ''Development of professional expertise:  Toward measurement of expert performance and design of optimal learning environments.''
 +
 
 +
===Publications on PLLC experiments===
 +
* Connelly, J. & Katz, S. (2006).  Intelligent dialogue support for physics problem solving:  Some preliminary mixed results.  ''Technology, Instruction, Cognition, and Learning,'' 4, 1-29.
 +
* Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), ''Intelligent Tutoring Systems: 8th International Conference, ITS2006.'' pp. 625-634. Amsterdam: IOS Press. 
 +
* Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp. 167-172 Mahwah, NJ: Erlbaum.
 +
* Chi, Min & VanLehn, K.  (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.''  pp. 548-550. Amsterdam, Netherlands: IOS Press.  
 +
* Chi, Min & VanLehn, K.  (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.'' pp. 551-553.  Amsterdam, Netherlands: IOS Press.
 +
* Chi, Min & VanLehn, K.  (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.''  pp. 409-416.  Amsterdam, Netherlands: IOS Press.
 +
* Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.''  pp. 554-556. Amsterdam, Netherlands: IOS Press.  
 +
* Hausmann, R. G. M. & VanLehn, K. (2007).  Explaining self-explaining:  A contrast between content and generation.  In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.'' pp. 417-424. Amsterdam, Netherlands: IOS Press.  
 +
* Hausmann, R. G. M. & VanLehn, K. (2007).  Self-explaining in the classroom:  Learning curve evidence  In D. McNamara & G. Trafton (Eds.) ''Proceedings of the 29th Annual Conference of the Cognitive Science Society.'' pp 1067-1072 Mahwah, NJ: Erlbaum.
 +
* Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), ''Artificial Intelligence in Education 2007''.
 +
* Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A.  (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.)  ''Artificial Intelligence in Education.'' pp 246-253. Amsterdam, Netherlands: IOS Press.
 +
* VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step?  Toward general, abstract representations of tutoring system log data.  In C. Conati & K. McCoy (eds).  ''Proceedings of User Modelling 2007.''
 +
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
 +
* Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
 +
* Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
 +
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Gaming Behaviors in an Intelligent Tutoring System. Proceedings of the International Conference on Intelligent Tutoring Systems, 2010, Pittsburgh, 184-193. [http://www.springerlink.com/content/92mru151006571q3/]
 +
* Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010).  An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System: Predictors and Impacts.  Journal of User Modeling and User Adapted Interaction, Special Issue on Educational Data Mining.  [http://www.springerlink.com/content/j33110w75053463l/]
  
 
==Current Status==
 
==Current Status==
  
−
Our main contact at the US Naval Academy (USNA), Don Treacy, has retired at the end of Spring 2007.  Another instructor, Mary Wintersgill, has agreed to take over as our main USNA contact.  The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks.  Raw log data from Andes is captured by OLI and transferred to the [http://learnlab.web.cmu.edu/ DataShop].  Other data, such as hardcopies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments and stored in locked file cabinets or secure servers.  All data is anonymized, only the instructors and one of the Andes developers at Pitt have the mapping with students’ identities.
+
As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College.
 +
At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan.  At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors.  The instructor at St. Anselm College is David Guerra, who generally
 +
teaches one section.
 +
The students use [http://www.cmu.edu/oli/ Open Learning Initiative (OLI)] to access [[Andes]], and the instructors use OLI to view gradebooks.  Both high school and college students use Andes at home to do their regular homework assignments.  Occasionally, Andes is used in class, but such “seat work” is not common.
  
−
As of fall 2007, [[Andes]] is being used at the following institutions:
+
Raw log data from Andes is stored on OLI servers.  The raw data is periodically converted to [http://learnlab.web.cmu.edu/ DataShop] format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data.  Researchers thus refer to both types of data.
−
* St. Anselm college, Manchester NH (1 instructor).
+
 
−
* US Naval Academy (2 instructors, several sections).
+
All user identification is encrypted.  The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein.  Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers. 
−
* Watchung Hills Regional High School, Warren NJ (2 instructors).
+
 
 +
Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies.   For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.
  
 
==Plans==
 
==Plans==
  
−
Our major goal continues to be to expand the number of sites and instructors involved in the PLLC.  There are simply not enough lab slots and students to meet the existing demand from PLLC experimenters.
+
Our major goal is to expand the number of instructors and researcher involved with the PLLC.
−
In addition, our most supportive instructor at the US Naval Academy, Don Treacy, has recently retired.   
+
In order for this to happen, we need to increase the number of instructors using Andes.
−
In order to increase involvement in the PLLC, we first need to increase the number of instructors using Andes in their courses.  So far, we have had limited success in doing this.  Instructors have been reluctant to adopt Andes for the following reasons:  instructors want a homework system to cover their entire course;  they want Andes problems to fit their instructional style; they want any hints given to be effective; and they want reasonable student actions to be accepted.  Finally, Andes is not well-known in the general physics community.
+
Also, we believe there is great mutual benifit in working closely with members of the [http://www.compadre.org/per/ Physics Education Research] community.
 +
 
 +
===Increase awareness of Andes===  
 +
We need to increase awareness of Andes in the physics community.
 +
To date, we have focused our efforts on national meetings of
 +
the AAPT and APS.  However, we plan to broaden our efforts:
 +
* We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future. 
 +
* We plan to arrange a summer school targeted mainly at regional high-school teachers of physics.  Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
 +
* Continue visiting physics departments at other universities.
 +
* Publish PLLC-related research in the physics education journals.
 +
 
 +
===Improvements to Andes itself===
 +
Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:
 +
* Instructors want Andes to be a commercial product.  In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional.  Currently, Andes is available thought the open source Learning Management System [http://www.lon-capa.org LON-CAPA].  We continue to work on our goal of offering Andes through our commercial partner, [http://www.webassign.net WebAssign].
 +
* Instructors want all reasonable student actions to be accepted.  We need to expand the ability of Andes to understand a wide variety of problem solutions.
 +
* Instructors want good, effective hints.  We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools.  However, to really improve the hint quality would require that Andes maintain a model of the student across problems.  This is one aspect of expert human tutoring that we can't capture with the existing system.
 +
* Other improvements requests that we hear regularly:
 +
** Allow sensitivity to lengths of vectors.
 +
** Allow vector equations (currently, Andes equations are all scalar).
 +
** Instructor control over policy for student actions that are correct but don't contribute to a solution.
  
 +
===Grading policy===
 +
Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores.
 +
There are two problems:
 +
* We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
 +
* Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice.  For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.
 +
Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.
  
−
Because we largely achieved the first long-term goal of complete course coverage, we are shifting the focus of our attention to increasing the awareness of Andes in the physics teaching community, increase the quality of hints provided in Andes, improve our quality control process, and provide additional course content requested by new physics instructors.
+
===Supporting existing Andes users=== 
−
+
There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.
 +
* Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
 +
* Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
 +
* Add instructor requested problem types (such as graph drawing).
 +
* Fixing instructor reported bugs and complaints promptly.  In particular, Andes sometimes gives hint sequences that are not helpful.  Also, it sometimes won't accept solution steps that instructors would allow.
 +
* Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
 +
* Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.
  
−
* '''Increasing Andes awareness in the physics community'''
+
===Log file analysis===
−
**Present talks and posters at the American Association of Physics Teachers conference in January 2008 and the American Physics Society (APS) meeting in April 2008.
+
The [[knowledge component]]s (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected.  In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts.  Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores.  We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.
−
**Continue to visiting physics departments at other universities.
 
−
** Publish Andes-based research in the physics education journals.
 
−
* '''Supporting existing Andes instructors''':  There are a number of non-PLLC instructors using Andes in their classrooms.  We need to make their experience with Andes a positive one.  Hopefully, a positive experience will lead to interest in participating in LearnLab.  This includes:
 
−
**Adding instructor-requested homework problems
 
−
**Fixing instructor-reported bugs promptly, and
 
−
**Including some instructor control over the hinting behavior of Andes.
 
  
−
* '''Increasing instructor acceptance''':  In order to increase the number of instructors using Andes, we need to address the following issues:
+
Here are some continuing activities associated with log files:
−
** '''Improving hints given to students''':  [[Andes]] is supposed to mimic the hints that an expert (human) tutor would give to students.  However, sometimes the hints are misleading or do not help the student better understand the problem at a deep level.
+
* Download log files from OLI, anonymize them, and load them into the DataShop.
−
** '''Non-obvious conventions''':  Communicating physics and math precisely and unambigously entails using some notational and user interface conventions that user find non-obvious.  We call these non-obvious conventions (NOCs; pronounced “knocks”).  We need to collect as many non-obvious conventions as we can think of and fix them or highlight them in the videos and other training.  
+
* Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
−
** '''Lesser priority items''':  There are a number of improvements to Andes that would lead to increased instructor happiness.
+
* Finish investigating whether time spent can be a useful metric of student learning.
−
*** '''Vectors in equations''':  Handling of equivalent variables.  And the equations that use them.
+
* Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
−
*** '''Scale drawing of vectors.'''
+
* Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.
−
*** '''True but irrelevant entries''':  Currently these turn red.  Instructor should be able to select the color and/or the warning that appears.  They should be able to have separate policies for equations and non-equations.
 

Latest revision as of 21:07, 6 September 2011

Physics LearnLab Course

The Physics LearnLab Course (PLLC) is a research facility for studying how students learn introductory physics. It provides baseline data on student activities throughout the physics course, and it hosts specific research studies that measure the improvement in students’ learning caused by changes in the instruction. At this time (2010), it is sited in:

  • the two-semester Introductory Physics courses at the US Naval Academy in Annapolis, MD
  • three courses at Watchung Hills Regional High School in Warren, NJ.
  • one-semester introductory courses at St. Anselm College in Manchester, New Hampshire.


Students in PLLC classes use the Andes intelligent tutoring system to do their homework. Andes allows the PLLC to collect fine-grained data on student activity through the entire semester. The remainder of the course is taught the usual way, with lectures, labs, and a commercial paper-based textbook. In vivo experiments take place either by modifying Andes or by running studies during lab sessions that instructors have “donated” to the PLLC.

Studies Conducted

Completed studies:

  • Comparing two homework systems, Sophia Gershman, 2008-2009.
  • Analogical Scaffolding in Collaborative Learning, lab study (Gadgil &Nokes, 2009)
  • An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System. Data mining study (Kasia Muldner, Winslow Burleson, Brett Van de Sande, Kurt VanLehn, 2009-2010).

In progress or planned:

  • Comparing two homework systems, Sophia Gershman, 2010-2011.
  • Reflective dialogs (Sandy Katz, Diane Litman, & Pam Jordan, 2010-2013).

Achievements

From its inception in January 2005 to the present, we have achieved the following:

Content development milestones

  • The number of Andes problems assigned by instructors at the Naval Academy has increased from 58% to 100% in the Fall semester, and from 42% to 75% in the Spring semester.
  • We have increased the total number of working Andes problems from 350 to 556.
  • The number of physics principles has increased from 126 to 219. The number of rules in the physics “Knowledge Base” (the AI system) has increased from 619 to 915. The number of scalar quantities defined in Andes has increased from 85 to 126.
  • We shot videos of problems being solved—at least one per problem set—and revised many of the older videos. These act as worked examples. Students who view the videos in a problem set before solving any problems have a much easier time of it.

Enabling Technologies

  • We developed a way to run Andes under OLI. In particular, we found ways to get them to communicate through the USNA firewall, to upload log data and solution files, and to recover gracefully from most crashes.
  • We developed a method to control the data that the OLI gradebook exports to spreadsheets so that only the data that instructors wanted was exported in a format they specified.
  • Implemented “gating,” a method to force students to solve Andes problems in a pre-determined order. This was needed for the Sandy Katz experiment in fall 2006.
  • Andes raw logs can now be converted to the [DataShop] format at the knowledge component level (June 2007). The knowledge components associated with each correct student action (corresponding with a step) and must incorrect action (see transaction) is determined by Andes.

Log file analysis

The Andes log files represent a rich source of information about student problem solving but have not been studied in depth, outside the needs of specific experiments. We have begun to study the log files and begun to promote such work in the Physics Education Research (PER) community.

  • Studied time usage (how long does it take to apply a KC?) and time-on-task (are they really working?). Investigated whether time-on-task could be used as a metric for student learning of KC's.
  • Begun comparing the Log data to end-of-semester surveys administered at the USNA. The surveys were not anonymous, so individual survey results can be matched with the associated log files.
  • Conducted a workshop on log file analysis at PERC 2007. Two senior members of the PER community, Joe Redish and Gerd Kortemeyer, attended, expressed initial interest and corresponded with us after the conference, but no firm plans have been made.

Adoption of Andes

As of Spring 2010, Andes is being used at the following institutions:

  • St. Anselm college, Manchester NH (1 instructor).
  • US Naval Academy (1 instructor, several sections).
  • Upper St. Clair School District, PA (1 instructor).
  • Clark County School District, NV (1 instructor, distance learning).
  • Clark College, WA (1 instructor, several sections)
  • Watchung Hills Regional High School, Warren NJ (2 instructors, several sections).

We observe steadily growing use of Andes by individuals not enrolled in any OLI course. During Spring 2010, between 285 and 1332 different users (some use is anonymous, precluding an exact count) solved a total of 15192 Andes problems. The previous semester, a total of 11819 problems were solved.

Advertising Andes in the physics community

We have focused our efforts on meetings of the American Association of Physics Teachers (AAPT) and the American Physical Society (APS) where we have presented numerous talks, posters, and a workshop.

These meetings generally do not publish proceedings.

More recently, we have begun promoting the Physics LearnLab at regional AAPT meetings:

  • Touring the Electromagnetic Spectrum (OSAPS 2008), Youngstown OH, March 2008. Vendor exhibit.
  • Central Pennsylvania Section of the American Association of Physics Teachers (CPS/AAPT), Lock Haven PA, April 2008. Vendor exhibit.
  • Fall meeting of the Arizona section of the AAPT, October 2008. Workshop for instructors.

In addition, we have presented Andes at other universities: Southern Methodist University (2006), the Ohio State University (2007), Rutgers University (2007), US Air Force Academy (2007), and the US Naval Academy (2007).

Publications on Andes

  • VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. The Andes Physics Tutoring System: Lessons Learned. International Journal of Artificial Intelligence and Education, 15 (3), 1-47.
  • Vanlehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D. J., Weinstein, A., and Wintersgill, M. C. The Andes physics tutoring system: Five years of evaluations. In G. McCalla, C. K. Looi, B. Bredeweg & J. Breuker (Eds.), Artificial Intelligence in Education. (pp. 678-685) Amsterdam, Netherlands: IOS Press.
  • Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp 246-253. Amsterdam, Netherlands: IOS Press.
  • VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). Proceedings of User Modelling 2007.
  • VanLehn, K., & van de Sande, B. (in press) Expertise in elementary physics, and how to acquire it. In K. A. Ericsson (Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments.

Publications on PLLC experiments

  • Connelly, J. & Katz, S. (2006). Intelligent dialogue support for physics problem solving: Some preliminary mixed results. Technology, Instruction, Cognition, and Learning, 4, 1-29.
  • Ringenberg, M. & VanLehn, K. (2006). Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In K. Ashley & M. Ikeda (Eds.), Intelligent Tutoring Systems: 8th International Conference, ITS2006. pp. 625-634. Amsterdam: IOS Press.
  • Chi, Min & VanLehn, K. (2007) The impact of explicit strategy instruction on problem-solving behaviors across intelligent tutoring systems. In D. McNamara & G. Trafton (Eds.) Proceedings of the 29th Annual Conference of the Cognitive Science Society. pp. 167-172 Mahwah, NJ: Erlbaum.
  • Chi, Min & VanLehn, K. (2007) Domain-specific and domain-independent interactive behaviors in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 548-550. Amsterdam, Netherlands: IOS Press.
  • Chi, Min & VanLehn, K. (2007) Porting an intelligent tutoring system across domains. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 551-553. Amsterdam, Netherlands: IOS Press.
  • Chi, Min & VanLehn, K. (2007) Accelerated future learning via explicit instruction of a problem solving strategy. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 409-416. Amsterdam, Netherlands: IOS Press.
  • Craig, S. D., VanLehn, K., Gadgil, S., & Chi, M. T. H. (2007). Learning from collaboratively observing videos during problem solving with Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 554-556. Amsterdam, Netherlands: IOS Press.
  • Hausmann, R. G. M. & VanLehn, K. (2007). Explaining self-explaining: A contrast between content and generation. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp. 417-424. Amsterdam, Netherlands: IOS Press.
  • Hausmann, R. G. M. & VanLehn, K. (2007). Self-explaining in the classroom: Learning curve evidence In D. McNamara & G. Trafton (Eds.) Proceedings of the 29th Annual Conference of the Cognitive Science Society. pp 1067-1072 Mahwah, NJ: Erlbaum.
  • Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger & J. Greer (Eds.), Artificial Intelligence in Education 2007.
  • Nwaigwe, A., Koedinger, K.,VanLehn, K., Hausmann, R. G. M. & Weinstein, A. (2007) Exploring alternative methods for error attribution in learning curves analyses in intelligent tutoring systems. In R. Luckin, K. R. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education. pp 246-253. Amsterdam, Netherlands: IOS Press.
  • VanLehn, K., Koedinger, K., Skogsholm, A., Nwaigwe, A., Hausmann, R.G.M., Weinstein, A. & Billings, B. (2007). What’s in a step? Toward general, abstract representations of tutoring system log data. In C. Conati & K. McCoy (eds). Proceedings of User Modelling 2007.
  • Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, May). Trialog: How Peer Collaboration Helps Remediate Errors in an ITS. Paper presented at the 21st meeting of the International FLAIRS Conference, Coconut Grove, FL.
  • Hausmann, R. G. M., van de Sande, B., & VanLehn, K. (2008, June). Shall we explain? Augmenting Learning from Intelligent Tutoring Systems and Peer Collaboration. Paper presented at the 9th meeting of the International Conference on Intelligent Tutoring Systems, Montréal, Canada.
  • Hausmann, R. G. M., van de Sande, B., van de Sande, C., & VanLehn, K. (2008, June). Productive Dialog During Collaborative Problem Solving. Paper presented at the 2008 International Conference for the Learning Sciences, Utrecht, Netherlands.
  • Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Gaming Behaviors in an Intelligent Tutoring System. Proceedings of the International Conference on Intelligent Tutoring Systems, 2010, Pittsburgh, 184-193. [1]
  • Muldner, K., Burleson, B., van de Sande, B., VanLehn, K. (2010). An Analysis of Students’ Gaming Behaviors in an Intelligent Tutoring System: Predictors and Impacts. Journal of User Modeling and User Adapted Interaction, Special Issue on Educational Data Mining. [2]

Current Status

As of Spring 2010, the PLLC consists of courses at the US Naval Academy, Watchung Hills Regional High School in New Jersey, and St. Anselm College. At the US Naval Academy, 3 sections of 25 students each taught by Professor Ted McClanahan. At Watchung Hills Regional High School, the instructors are Sophie Gershmann and Brian Brown who teach three different levels of physics courses, mostly for Juniors and Seniors. The instructor at St. Anselm College is David Guerra, who generally teaches one section. The students use Open Learning Initiative (OLI) to access Andes, and the instructors use OLI to view gradebooks. Both high school and college students use Andes at home to do their regular homework assignments. Occasionally, Andes is used in class, but such “seat work” is not common.

Raw log data from Andes is stored on OLI servers. The raw data is periodically converted to DataShop format, but the conversion process is still not completely satisfactory, as some information is still available only from the raw log data. Researchers thus refer to both types of data.

All user identification is encrypted. The mapping between encrypted identities and student names is held by the Andes development programmer, Anders Weinstein. Instructors see only the students’ user identification before encryption; researchers see only the encrypted identities. Non-log data, such as hard-copies of midterm exams or audio files from verbal protocols, are collected as needed for specific experiments. They are anonymized by Anders Weinstein and stored in locked file cabinets or secure servers.

Although most experiments are in vivo experiments conducted in the PLLC courses, some studies are conventional lab studies. For instance, an experimenter might first run a study in the lab with paid volunteers and later do an improved version of the study in one or more PLLC classes.

Plans

Our major goal is to expand the number of instructors and researcher involved with the PLLC. In order for this to happen, we need to increase the number of instructors using Andes. Also, we believe there is great mutual benifit in working closely with members of the Physics Education Research community.

Increase awareness of Andes

We need to increase awareness of Andes in the physics community. To date, we have focused our efforts on national meetings of the AAPT and APS. However, we plan to broaden our efforts:

  • We have begun to promote Andes at regional AAPT meetings and hope to expand this effort in the future.
  • We plan to arrange a summer school targeted mainly at regional high-school teachers of physics. Our long-term desire is for the summer school activity to eventually grow into a community of users consisting of both high school and college level instructors.
  • Continue visiting physics departments at other universities.
  • Publish PLLC-related research in the physics education journals.

Improvements to Andes itself

Based on conversations with potential instructors as they view demonstrations of Andes and on instructors who have dropped Andes after using it, we have identified several aspects of Andes itself that we need to improve:

  • Instructors want Andes to be a commercial product. In particular, they are worried about the long-term stability of the software product and that user support may be sporadic or unprofessional. Currently, Andes is available thought the open source Learning Management System LON-CAPA. We continue to work on our goal of offering Andes through our commercial partner, WebAssign.
  • Instructors want all reasonable student actions to be accepted. We need to expand the ability of Andes to understand a wide variety of problem solutions.
  • Instructors want good, effective hints. We plan to make instructor evaluations of hint sequences an integral part of future workshops and summer schools. However, to really improve the hint quality would require that Andes maintain a model of the student across problems. This is one aspect of expert human tutoring that we can't capture with the existing system.
  • Other improvements requests that we hear regularly:
    • Allow sensitivity to lengths of vectors.
    • Allow vector equations (currently, Andes equations are all scalar).
    • Instructor control over policy for student actions that are correct but don't contribute to a solution.

Grading policy

Unfortunately, the current grading rubric is opaque and complicated and we are not always happy with the validity of the scores. There are two problems:

  • We don’t have any mechanism for an instructor to understand or modify the scoring rubric.
  • Some students become focused on raising their scores and, due to various weaknesses of (or incorrect inferences about) the scoring rubric, engage in behaviors that may raise their scores but do not constitute good problem-solving practice. For instance, a student will put in the final answer to a problem, and then go back and add problem-solving steps until their score is acceptably high.

Since one of the main goals of a grading policy is to encourage students to engage in productive problem solving behavior, any changes to the grading policy must be accompanied by log file analysis.

Supporting existing Andes users

There are a number of non-PLLC instructors using Andes in their classrooms as well as a number of users not affiliated with any OLI course.

  • Provide instructor support for setting up and running classes and user support for difficulties installing and running Andes.
  • Add instructor requested homework problems. We will continue our policy of adding new content based on instructor requests.
  • Add instructor requested problem types (such as graph drawing).
  • Fixing instructor reported bugs and complaints promptly. In particular, Andes sometimes gives hint sequences that are not helpful. Also, it sometimes won't accept solution steps that instructors would allow.
  • Develop log file analysis to detect ineffective hint sequences, common student difficulties, and plain old bugs.
  • Eventually, hold some instructor workshops for existing instructors, so that they feel part of the Andes development process and connect with other Andes users.

Log file analysis

The knowledge components (KCs) used by Andes generally do not produce the nice learning curves that one would expect, which makes it problematic for experimenters to use them as dependent measures. We suspect that the present physics KCs implicitly contain the knowledge needed for applying a principle within a problem context along with the principle itself. Thus, when a KC that has been practiced several times in simple problems is used for the first time in a complex problem, the associated assistance score may be higher than expected. In fact, it is common practice in physics homework assignments to exercise students in applying physics principles in widely varying problem contexts. Thus, as the problem context varies, the difficulty of applying our present KCs vary widely, resulting in widely varying assistance scores. We have been doing data mining to test this hypothesis, but this has been a backburner activity and is moving slowly.

Here are some continuing activities associated with log files:

  • Download log files from OLI, anonymize them, and load them into the DataShop.
  • Andes raw logs can be converted to the DataShop format, but the converted logs often do not have the right information in them for the kinds of analysis experimenters want to do, so the converter scripts must be changed.
  • Finish investigating whether time spent can be a useful metric of student learning.
  • Continue investigating why the present KC’s produce learning curves that do not match current theoretical predictions.
  • Continue promoting Log file analysis as an interesting area of research, especially for those interested in developing cognitive models of student learning.
Retrieved from "https://learnlab.org/wiki/index.php?title=Physics&oldid=12232"