Difference between revisions of "Educational Research Methods 10"

From LearnLab
Jump to: navigation, search
(Class URL:)
(Undo revision 12311 by Koedinger (Talk))
 
(98 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 +
(For Spring 2012 go to [[Educational Research Methods 2012]])
 +
 
==Research Methods for the Learning Sciences 85-748==
 
==Research Methods for the Learning Sciences 85-748==
Spring 2008 Syllabus Carnegie Mellon University
+
Spring 2010 Syllabus Carnegie Mellon University
 
   
 
   
====Class times:====
+
====Class times====
4:30 to 5:50 Tuesday & Thursday in NSH 3001
+
4:30 to 5:50 Tuesday & Thursday
  
====Location:====
+
====Location====
3501 NSH
+
336B Baker Hall for the first day.
 +
 
 +
3501 Newell Simon Hall thereafter.
 +
 
 +
====Instructors====
 +
Professor Ken Koedinger
  
====Instructors:====
 
Professor Kenneth R. Koedinger
 
Office hours by appointment
 
 
Location: 3601 Newell-Simon Hall
 
Location: 3601 Newell-Simon Hall
 +
 
Phone: 8-7667
 
Phone: 8-7667
 +
 
Email: Koedinger@cmu.edu
 
Email: Koedinger@cmu.edu
 +
 +
Office hours by appointment
 +
 
   
 
   
 
Dr. Philip I. Pavlik Jr.
 
Dr. Philip I. Pavlik Jr.
Office hours by appointment
+
 
 
Location: 300S Craig St, 224
 
Location: 300S Craig St, 224
 +
 
Phone: 8-1618
 
Phone: 8-1618
 +
 
Email: ppavlik@andrew.cmu.edu
 
Email: ppavlik@andrew.cmu.edu
  
====Class URL:====
+
Office hours by appointment
[http://www.learnlab.org/research/wiki/index.php/Educational_Research_Methods_09 Main page]
+
 
 +
=====Teaching Assistant=====
 +
Benjamin Shih
 +
 
 +
Location: GHC 8003
  
====Goals:====
+
Phone: 8-6289
 +
 
 +
Email: shih@cmu.edu
 +
 
 +
Office hours by appointment
 +
 
 +
====Class URL==== 
 +
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]
 +
 
 +
[https://wave.google.com/wave/ Google wave (see your email for login information or contact TA)]
 +
 
 +
====Goals====
 
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education.  The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis.  A key goal is to help students think about and learn how to apply these methods to their own research programs.
 
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education.  The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis.  A key goal is to help students think about and learn how to apply these methods to their own research programs.
  
====Course Prerequisites:====
+
====Course Prerequisites====
 
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.   
 
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.   
  
====Readings Textbook:====  
+
====Textbook and Readings====  
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly
+
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.  You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]
Other readings will be assigned in class.
+
 
+
Other readings will be assigned in class.
====Reading Reports:====
+
 
Students are required to submit at least two posts per week to the course discussion group/blog before class, on either Sunday/Monday/Tuesday before class (for readings due Tuesday) or on Tuesday/Wednesday/Thursday before class (for readings due Thursday). These posts can be about
+
====Reading Reports====
#a question you had about the reading, something important you did not understand
+
We will be using Google Wave for course reading reports and discussions.  Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time.  Further details and account invitations will be discussed in class.
#an idea inspired by the reading
 
#an interesting connection with something you learned or did previously in this or another course, or in other professional work or research
 
#an on-topic, relevant response, clarification, or further comment on another student’s post
 
  
====Grading:====
+
Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.
In order to make the grading as systematic and equitable as possible, we are using a point system in which each homework assignment and the exams are given a certain number of points. Points associated with each assignment are listed below.  
 
  
{| class="wikitable"
+
{| class="wikitable" border="1"
|-
 
! Assignment
 
!
 
! Points
 
 
|-
 
|-
| Homework (5 parts)
+
!  Monday
|
+
!  Tuesday
| 270
+
!  Wednesday
 +
!  Thursday
 +
!  Friday
 +
!  Saturday
 +
!  Sunday
 
|-
 
|-
| 1. Tutor Evaluation
+
| Original Post (Tuesday Reading)
| 30
+
| Reply (Tuesday Reading)
|
+
Summary Edits (Tuesday Reading)
|-
+
|
| 2. Addition Pseudo Tutor
+
| Original Post (Thursday Reading)
| 40
+
Summary Edits (Thursday Reading)
|
+
|
|-
+
|
| 3. Tutor Modification Task1
+
| Reply (Thursday Reading)
| 40
 
|
 
|-
 
| 4. Tutor Modification Task2(1)
 
| 70
 
|
 
|-
 
| 5. Think Aloud & Difficulty Factors Assessment(1)
 
| 90
 
|  
 
|-
 
| Final Project
 
|
 
| 160
 
|-
 
| Mid-term Exam
 
|
 
| 200
 
|-
 
| Final Exam
 
|
 
| 200
 
|-
 
| Reading Reports
 
|
 
| 120
 
|-
 
| Instructor's Judgement
 
|
 
| 50
 
|-
 
| Total
 
|
 
| 1000
 
 
|}
 
|}
# These are pair or group assignments.
 
# Note, 5% of your course grade will be based on the instructors’ judgment of your level of learning , whether you have learned more or less than is  reflected in the points you have earned. This judgment will be influenced by your class and group participation. Thus, regular attendance and active participation in group and class meetings is in your interest.
 
  
Group homework grades are closely tied to individual grade. For example, if the group grade is 25 points, each individual member is likely to receive 25 points. However, to avoid rewarding free riding, distinctions will be made among individuals. We will not have adequate personal information to distinguish among group members for group work. Therefore in assigning individual grades, private evaluations from all group members will be solicited at the end of the course. These evaluations will take the form of a questionnaire asking each group member to allocate, among 100% of the total effort, the responsibility among the members for the group's homework assignments.
+
=====Posts and Replies=====
 +
 
 +
Original posts should contain at least one of the following:
 +
*a question you had about the reading or something important you did not understand
 +
*an idea inspired by the reading
 +
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research
 +
 
 +
For readings due on a Tuesday, the original post must be submitted by Monday morning.
 +
 
 +
For readings due on a Thursday, the original post must be submitted by Thursday morning.
 +
 
 +
Replies must be:
 +
*an on-topic, relevant response, clarification, or further comment on another student’s post
 +
 
 +
For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.
 +
 
 +
For readings due on a Thursday, at least one reply must be submitted by Sunday morning.
 +
 
 +
This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after.  Please use this extra time to have a full and meaningful discussion on the topics discussed.
 +
 
 +
=====Summary=====
 +
 
 +
For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.
 +
 
 +
Each summary will consist of:
 +
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter.  For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
 +
* A brief discussion of the methodology.  For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed.  For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
 +
* A listing of major issues or suggestions for the paper, as related to the course.  Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.
 +
 
 +
The first two parts of the summary should be complete by the morning of the day of class.
 +
 
 +
====Grading====
 +
 
 +
There will be assignments associated with each section of the course.  Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.
 +
 
 +
* Course work
 +
** 10% Reading reports 
 +
** 50% Homework assignments
 +
* Project & final paper
 +
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.
 +
::# Apply a method from the class to your research. You should not choose a method that you have a large amount of experience with, but you may choose a method that you were planning to use for some upcoming project anyway.
 +
::# Think of it as writing a grant proposal, and remember it could actually form part of a grant if you so choose. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method.  But, please check with us to get feedback and approval on a proposed change.
 +
::# No more than 10 double-spaced pages -- Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please beginning with a bit of literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data. Certainly you will want to account for any challenges to validity that seem plausible as part of your design and writeup.
 +
 
 +
::::* Final Project Milestones
 +
:::::March 5 - a paragraph project proposal in email
 +
:::::March 15 - approval of proposed project
 +
:::::April 20 - turn in a draft for feedback
 +
:::::April 26 - feedback provided by April 26
 +
::::::Desired: set up a meeting with one of the faculty to discuss draft
 +
:::::May 5 - final paper due
 +
 
 +
====Class Schedule====
 +
 
 +
=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
 +
*1-12-10
 +
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
 +
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
 +
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
 +
*1-14-10
 +
**Reading: Trochim Ch 1 and 7
 +
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
 +
*1-19-10
 +
**Reading: Trochim Ch 9 and 10
 +
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
 +
*1-21-10
 +
**Reading: Trochim Ch 11
 +
**Assignment 1 due before class
 +
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
 +
 
 +
=====Cognitive Task Analysis (Koedinger, Pavlik) =====
 +
*1-26-10
 +
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
 +
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
 +
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]
 +
 
 +
*1-28-10
 +
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.
 +
 
 +
**Heffernan, N. & Koedinger, K. R. (1997).  The composition effect in symbolizing: The role of symbol production vs. text comprehension:  In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.
 +
 
 +
*2-2-10
 +
**Go to PSLC poster session outside of 6115 Gates
 +
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
 +
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices
 +
 
 +
=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
 +
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
 +
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
 +
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007).  Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
 +
*2-9-10 Snowmagedon
 +
*2-11-10 Protocol Analysis of Educational Discussions
 +
**[Half the class will read this one]Veel, R. (1999).  Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_7.pdf]]
 +
**[Half the class will read this one] Williams, G. (1999).  The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_4.pdf]]
 +
**[Optional] Martin, J. and White, P. R. (2005).  The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Media:Martin-WhiteChapter_3.pdf]]
 +
** [Optional] Michaels et al., 2007 paper on Accountable Talk [[Media:2007_Deliberative_Discourse.pdf‎|AccountableTalkPaper]]
 +
*2-16-10
 +
** van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press.  Chapter 7[[Media:VanSch7.pdf‎]]
 +
**Kumar, R., Ai, H., and Rosé (submitted).  Choosing Optimal Levels of Social Interaction – Towards creating Human-like Conversational Tutors, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010-thermo-rk.pdf|ITS2010-Kumar]]
 +
** Data set from Kumar et al. study [[Media:Thermo-f09-transcripts.xls‎|Data]]
 +
** Iris's coding manual [[Media:Thermo08CodingScheme-v9b.doc|Manual]]
 +
** Chapter with alternative presentation of Reasoning coding [[Media:Sionti-Revised-Again.doc|Chapter]]
 +
*2-18-10
 +
** Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008).  Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[http://www.cs.cmu.edu/~cprose/pubweb/Proof2.pdf]]
 +
** Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010_submission_172.pdf]]‎
 +
** Hua Ai's Lecture [[Image:EduMethod.ppt]]
 +
*2-23-10
 +
** Schooler, J. W., Ohlsson, S., & Brooks, K. (1993).  Thoughts Beyond Words: When Language Overshadows Insight, Journal of Experimental Psychology 122(2), pp 166-183. [[Media:Schooleretal.pdf‎]]
 +
*2-25-10
 +
** Download SIDE and the SIDE User's Manual from the webpage.  [[http://www.cs.cmu.edu/~cprose/SIDE.html]]
 +
 
 +
=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
 +
*3-2-10
 +
 
 +
1. From Trochim:
 +
 
 +
  A. Chapter 3 - the vocabulary of measurement
 +
         
 +
  B. Chapter 5 - on constructing scales (it's ok to focus
 +
      on the material up through sect 5.2a; the rest is
 +
      more of a skim [but I'd be happy to talk about that
 +
      in class also])
 +
 
 +
2. On item response theory (IRT), a set of statistical models that are used
 +
to construct scales and to derive scores from them, especially in education
 +
and psychological research:
 +
 
 +
  A. [[Media:Harris-article.pdf|Harris Article (PDF)]]
 +
 
 +
  Please take and self-score the test at the end of
 +
  this article.  Count each part of question one as
 +
  one point, and each of the remaining three questions
 +
  as one point (no partial credit!).  Bring your 8
 +
  scores to class.  E.g. if you missed 1(c) and (d), and
 +
  you also missed question 4, then you would bring to
 +
  class the following scores:
 +
 
 +
  1 1 0 0 1 1 1 0
 +
 
 +
  If you missed 1(a) and (b) and question 2, bring the
 +
  following scores:
 +
 
 +
  0 0 1 1 1 0 1 1
 +
 
 +
  (note that the total score is 5 in both cases, but
 +
  the pattern of rights and wrongs differs; it is the
 +
  pattern that we are interested in).
 +
 
 +
  B. Please browse *online* through pp 1-23 of the pdf at
 +
  [http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].
 +
 
 +
  The math is a bit heavy going but there are links
 +
  to apps that illustrate various points in the
 +
  harris article. 
 +
 
 +
  So skim the math and play with the apps.
 +
 
 +
We will discuss on Tue whatever of this we can get through, and continue the
 +
discussion as needed on Thursday.  There will be additional readings for Thu
 +
to introduce fitting and using IRT models.
 +
 
 +
*3-4-10
 +
 
 +
1. Short introduction to R ([[Media:Rintro.pdf|Rintro.pdf]])
 +
   
 +
    Please download and install R for your computer (windows, mac or
 +
    linux) from [http://www.cran.r-project.org/].
 +
   
 +
    Then try all the things in sections 1-9 of this handout (section
 +
    10 is optional).  You can try most things by copying from the pdf
 +
    and pasting into the R command window.
 +
   
 +
    You do not have to be completely done with this by the time we
 +
    meet for lecture 2, but you should aim to finish it soon
 +
    afterwards.
 +
 
 +
2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])
 +
   
 +
    Please read up through p 266 only.
 +
   
 +
    The math is a bit heavy going so please try to read around it to
 +
    see what the point of the article is. 
 +
   
 +
    We will try to look at some of the data in the article as examples
 +
    in lecture 2.
 +
 
 +
3. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])
 +
   
 +
    Read through p 18.  This is a more modern modern look at some of
 +
    the same issues that are addressed in Trochim's chapters.
 +
   
 +
    The remainder of this paper surveys various probabilistic models
 +
    for the "measurement model" portion of Mislevy's framework (Figure
 +
    1).  It is quite interesting but we will not pursue it.
 +
 
 +
=====NO CLASS – Spring break=====
 +
*3-9-10
 +
*3-11-10
 +
=====Psychometrics continued =====
 +
*3-16-10
 +
=====Surveys, Questionnaires, Interviews (Kiesler) =====
 +
*3-18-10
 +
Reading: Trochim Ch 4 and 5
 +
*3-23-10
 +
Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883.  [[Media:Tourangeau_SensitiveQuestions.pdf]]
 +
 
 +
Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum.  [[Media:Tourangeau_RememberingWhatHappened.pdf]]
 +
 
 +
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
 +
*3-25-10
 +
Ritter, F.E., & Schooler, L. J. (2001). The learning curve.  In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf]]
 +
 
 +
Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family:  Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf]]
 +
 
 +
'''Assignment:''' Due Thursday, 4/1.  [[Media:Learning-curve-assignment.doc | DOC]]
 +
 
 +
*3-30-10
 +
Register an account on DataShop ( http://www.pslcdatashop.org ) and watch [https://pslcdatashop.web.cmu.edu/about/case-studies/ken-geometry/index.html this video].
 +
 
 +
''Reading'':
 +
Cen, H., Koedinger, K. R., & Junker, B. (2006).  Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
 +
*4-1-10
 +
Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
 +
 
 +
Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]] No Summary Required.
 +
*4-6-10
 +
Do Unit 2 in the OLI course Empirical Research Methods
 +
-- go to: http://oli.web.cmu.edu/openlearning/
 +
-- go to Empirical Research methods (on left tab)
 +
-- click on Peek In
 +
-- complete Unit 2
 +
 
 +
Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials.  Journal of Educational Computing Research, 32, 1, 1-26.  [[Media:Scheines jecr revised.doc | PDF]]
 +
*4-8-10
 +
*4-13-10
 +
*4-15-10 NO CLASS – Spring Carnival
  
====Class Schedule: ====  
+
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
(Topics continue into blanks!)
+
*4-20-10 
 +
*4-22-10
 +
=====Wrap-up=====
 +
*4-27-10
 +
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1).  [[Media:2004 Barab Squire.pdf|PDF]]
  
*[[1-13-09 ERM]] Basic Research & Experimental Methods (Koedinger, Pavlik)
+
Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]
*[[1-15-09 ERM]]
+
*4-29-10
*[[1-20-09 ERM]]
 
*[[1-22-09 ERM]]
 
*[[1-27-09 ERM]] Cognitive Task Analysis (Koedinger, Pavlik)
 
*[[1-29-09 ERM]]
 
*[[2-3-09 ERM]] 
 
*[[2-5-09 ERM]] Video and Verbal Protocol Analysis (Lovett, Rosé)
 
*[[2-10-09 ERM]]
 
*[[2-12-09 ERM]]
 
*[[2-17-09 ERM]]
 
*[[2-19-09 ERM]]
 
*[[2-24-09 ERM]]
 
*[[2-26-09 ERM]] Ethnography & Design Experiments?
 
*[[3-3-09 ERM]]  Surveys, Questionnaires, Interviews (Kiesler)
 
*[[3-5-09 ERM]]  
 
*[[3-10-09 ERM]] NO CLASS – Spring break
 
*[[3-12-09 ERM]] NO CLASS – Spring break
 
*[[3-17-09 ERM]] Psychometrics, reliability, Item Response Theory (Junker, Koedinger)
 
*[[3-19-09 ERM]]
 
*[[3-24-09 ERM]]
 
*[[3-26-09 ERM]] Educational data mining (Scheines, Pavlik)
 
*[[3-31-09 ERM]]
 
*[[4-2-09 ERM]]
 
*[[4-7-09 ERM]]
 
*[[4-9-09 ERM]]
 
*[[4-14-09 ERM]]
 
*[[4-16-09 ERM]] NO CLASS – Spring Carnival
 
*[[4-21-09 ERM]] Cognitive Task Analysis - Revisited (Koedinger, Pavlik)
 
*[[4-23-09 ERM]]
 
*[[4-28-09 ERM]] Wrap-up
 
*[[4-30-09 ERM]] Wrap-up
 

Latest revision as of 11:14, 3 December 2011

(For Spring 2012 go to Educational Research Methods 2012)

Research Methods for the Learning Sciences 85-748

Spring 2010 Syllabus Carnegie Mellon University

Class times

4:30 to 5:50 Tuesday & Thursday

Location

336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

Instructors

Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment


Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

Teaching Assistant

Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

Class URL

learnlab.org/research/wiki/index.php/Educational_Research_Methods_10

Google wave (see your email for login information or contact TA)

Goals

The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

Course Prerequisites

To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

Textbook and Readings

"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160

Other readings will be assigned in class.

Reading Reports

We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

Monday Tuesday Wednesday Thursday Friday Saturday Sunday
Original Post (Tuesday Reading) Reply (Tuesday Reading)

Summary Edits (Tuesday Reading)

Original Post (Thursday Reading)

Summary Edits (Thursday Reading)

Reply (Thursday Reading)
Posts and Replies

Original posts should contain at least one of the following:

  • a question you had about the reading or something important you did not understand
  • an idea inspired by the reading
  • an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:

  • an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

Summary

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:

  • A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
  • A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
  • A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

Grading

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

  • Course work
    • 10% Reading reports
    • 50% Homework assignments
  • Project & final paper
    • 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.
  1. Apply a method from the class to your research. You should not choose a method that you have a large amount of experience with, but you may choose a method that you were planning to use for some upcoming project anyway.
  2. Think of it as writing a grant proposal, and remember it could actually form part of a grant if you so choose. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
  3. No more than 10 double-spaced pages -- Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please beginning with a bit of literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data. Certainly you will want to account for any challenges to validity that seem plausible as part of your design and writeup.
  • Final Project Milestones
March 5 - a paragraph project proposal in email
March 15 - approval of proposed project
April 20 - turn in a draft for feedback
April 26 - feedback provided by April 26
Desired: set up a meeting with one of the faculty to discuss draft
May 5 - final paper due

Class Schedule

Basic Research & Experimental Methods (Koedinger, Pavlik)
Cognitive Task Analysis (Koedinger, Pavlik)
  • 1-26-10
    • Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
    • Assignment 2: Assignment2.doc
    • Slides: CTA-01.pdf
  • 1-28-10
    • Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.
    • Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.
Video and Verbal Protocol Analysis (Lovett, Rosé)
  • 2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
    • Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. Media:E&SPreface.pdf
    • Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. Media:Gilhooly.pdf‎
  • 2-9-10 Snowmagedon
  • 2-11-10 Protocol Analysis of Educational Discussions
    • [Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. Media:PedagogyChapter_7.pdf
    • [Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. Media:PedagogyChapter_4.pdf
    • [Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. Media:Martin-WhiteChapter_3.pdf
    • [Optional] Michaels et al., 2007 paper on Accountable Talk AccountableTalkPaper
  • 2-16-10
    • van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7Media:VanSch7.pdf‎
    • Kumar, R., Ai, H., and Rosé (submitted). Choosing Optimal Levels of Social Interaction – Towards creating Human-like Conversational Tutors, submitted to the Intelligent Tutoring Systems ConferenceITS2010-Kumar
    • Data set from Kumar et al. study Data
    • Iris's coding manual Manual
    • Chapter with alternative presentation of Reasoning coding Chapter
  • 2-18-10
    • Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[1]]
    • Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems ConferenceMedia:Its2010_submission_172.pdf
    • Hua Ai's Lecture File:EduMethod.ppt
  • 2-23-10
    • Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts Beyond Words: When Language Overshadows Insight, Journal of Experimental Psychology 122(2), pp 166-183. Media:Schooleretal.pdf‎
  • 2-25-10
    • Download SIDE and the SIDE User's Manual from the webpage. [[2]]
Psychometrics, reliability, Item Response Theory (Junker, Koedinger)
  • 3-2-10

1. From Trochim:

  A. Chapter 3 - the vocabulary of measurement 
          
  B. Chapter 5 - on constructing scales (it's ok to focus
      on the material up through sect 5.2a; the rest is
      more of a skim [but I'd be happy to talk about that 
      in class also])

2. On item response theory (IRT), a set of statistical models that are used to construct scales and to derive scores from them, especially in education and psychological research:

  A. Harris Article (PDF)
  
  Please take and self-score the test at the end of 
  this article.  Count each part of question one as
  one point, and each of the remaining three questions 
  as one point (no partial credit!).  Bring your 8
  scores to class.  E.g. if you missed 1(c) and (d), and
  you also missed question 4, then you would bring to
  class the following scores: 
  
  1 1 0 0 1 1 1 0
  
  If you missed 1(a) and (b) and question 2, bring the 
  following scores: 
  
  0 0 1 1 1 0 1 1 
  
  (note that the total score is 5 in both cases, but
  the pattern of rights and wrongs differs; it is the
  pattern that we are interested in).
  
  B. Please browse *online* through pp 1-23 of the pdf at
  [3].
  
  The math is a bit heavy going but there are links 
  to apps that illustrate various points in the 
  harris article.  
  
  So skim the math and play with the apps.

We will discuss on Tue whatever of this we can get through, and continue the discussion as needed on Thursday. There will be additional readings for Thu to introduce fitting and using IRT models.

  • 3-4-10

1. Short introduction to R (Rintro.pdf)

   Please download and install R for your computer (windows, mac or
   linux) from [4].
   
   Then try all the things in sections 1-9 of this handout (section
   10 is optional).  You can try most things by copying from the pdf
   and pasting into the R command window.
   
   You do not have to be completely done with this by the time we
   meet for lecture 2, but you should aim to finish it soon
   afterwards.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma (Junker, Sijtsma (PDF))

   Please read up through p 266 only.
   
   The math is a bit heavy going so please try to read around it to
   see what the point of the article is.  
   
   We will try to look at some of the data in the article as examples
   in lecture 2.

3. "Psychometric Principles in Student Assessment" by Mislevy et al (Mislevy (PDF))

   Read through p 18.  This is a more modern modern look at some of
   the same issues that are addressed in Trochim's chapters.
   
   The remainder of this paper surveys various probabilistic models
   for the "measurement model" portion of Mislevy's framework (Figure
   1).  It is quite interesting but we will not pursue it.
NO CLASS – Spring break
  • 3-9-10
  • 3-11-10
Psychometrics continued
  • 3-16-10
Surveys, Questionnaires, Interviews (Kiesler)
  • 3-18-10

Reading: Trochim Ch 4 and 5

  • 3-23-10

Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. Media:Tourangeau_SensitiveQuestions.pdf

Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. Media:Tourangeau_RememberingWhatHappened.pdf

Educational data mining (Scheines, Pavlik, Koedinger)
  • 3-25-10

Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. Media:RittterSchooler01.pdf

Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA Media:AIED2007_EDM_Zhang_ld_transfer.pdf

Assignment: Due Thursday, 4/1. DOC

  • 3-30-10

Register an account on DataShop ( http://www.pslcdatashop.org ) and watch this video.

Reading: Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. PDF

  • 4-1-10

Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. Media:2000_roberts_pashler.pdf

Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. Media:GOF.doc No Summary Required.

  • 4-6-10

Do Unit 2 in the OLI course Empirical Research Methods

-- go to: http://oli.web.cmu.edu/openlearning/ 
-- go to Empirical Research methods (on left tab)
-- click on Peek In
-- complete Unit 2

Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. PDF

  • 4-8-10
  • 4-13-10
  • 4-15-10 NO CLASS – Spring Carnival
Cognitive Task Analysis - Revisited (Koedinger, Pavlik)
  • 4-20-10
  • 4-22-10
Wrap-up
  • 4-27-10

Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). PDF

Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. PDF

  • 4-29-10