Educational Research Methods 2012

From LearnLab
Revision as of 17:57, 16 February 2012 by Koedinger (talk | contribs) (Cognitive Task Analysis - Revisited (Koedinger))
Jump to: navigation, search

Research Methods for the Learning Sciences 05-748

Spring 2012 Syllabus Carnegie Mellon University

Class times

4:30 to 5:50 Tuesday & Thursday


3001 Newell Simon Hall


Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email:, Office hours by appointment

Class URLs

Syllabus and useful links:

For reading reports:


The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

Course Prerequisites

To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

Textbook and Readings

"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

Reading Reports and Pre-Class Preparation

For more effective class meetings, students are asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ( for this purpose.

Unless otherwise directed by instructors, students should make two posts on the readings before 9am on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your two posts may be original or in response to another post (one of both is nice).

  • Original posts should contain one or more of the following:
    • something you learned from the reading or slides
    • a question you have about the reading or slides or about the topic in general
    • a connection with something you learned or did previously in this or another course, or in other professional work or research
  • Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.
Pre-Class Assignments

You may be asked to do other activities before class, such as answer questions on-line using the Assistment system, parts of the an OLI course, or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.


There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

  • Course work
    • 30% Before-class preparation, including reading reports, and in-class participation
    • 40% Assignments
  • Project & final paper
    • 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
  1. Apply a method from the class to your research. You should not choose a method that you already know well.
  2. Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
  3. No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

Class Schedule in Brief

  • Course Intro & Formulating Good Research Questions: Jan 17 (T)
  • Cognitive Task Analysis 1: Jan 19, 24, 26 (RTR)
  • Video and Verbal Protocol Analysis: Jan 31, Feb 2,7,9,14,16 (TRTRTR)
    • Guest Instructor(s): Marsha Lovett & Carolyn Rose
  • Cognitive Task Analysis 2: Feb 21, 23 (TR)
  • Educational Measurement & Psychometrics: Feb 28, Mar 1, 6 (TRT)
    • Guest Instructor(s): Brian Junker
  • Educational Design Research: Mar 8 (R)
  • NO CLASS – Spring break, Mar 13, 15 (TR)
  • Surveys, Questionnaires, Interviews: Mar 20, 22 (TR)
    • Guest Instructor(s): Sara Kiesler
  • Educational data mining: March 27, 29, Apr 3, 5, 10, 12 (TRTRTR)
    • Guest Instructor(s): Richard Scheines (& Ken Koedinger)
  • Experimental Methods: Apr 17, 24, 26, May 1 (TTRT)
    • NO CLASS – Spring Carnival, Apr 19 (R)
  • Wrap-up: May 3 (R)

Class Schedule with Readings and Assignments

NOTE: This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

Course Intro & Formulating Good Research Questions (Koedinger)
Cognitive Task Analysis (Koedinger)
  • 1-19-12
    • Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. Zhu&Simon-1987.pdf
    • Slides: CTA-01.pdf
    • [Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). PNAS-1996-Zhu-Simon.pdf
  • 1-24-12
    • Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. Clarketal2007-CTAchapter.pdf
      • One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's recent PIER talk, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
    • Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. HowPeopleLearnCh2.pdf
      • Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
      • As you read the 1-24 and 1-26 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about that domain, how you might perform a CTA, and you could represent the output of your analysis.
  • 1-26-12
    • Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. AlevenITS2004.pdf
    • Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. Klahr&carver88.pdf
    • Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. Siegler76.pdf
      • Pick one of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to how the authors represent the output of their analysis: Do they use production rules? What kinds of diagrams do they use?
  • Other possible readings:
Video and Verbal Protocol Analysis (Lovett, Rosé)
  • 1-31-12: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
    • Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. Media:Gilhooly.pdf‎
    • Wang, H. C., Rosé, C. P., Chang, C. Y. (2011). Agent-based Dynamic Support for Learning from Collaborative Brainstorming in Scientific Inquiry, International Journal of Computer Supported Collaborative Learning 6(3), pp 371-396. (just read pages 1-7 and 11-14 of the pdf) Media:CameraReady-ijCSCL2011-BrainStorming.pdf
    • Discussion Questions:
      • What are the main research questions being addressed in these two articles?
      • Why did the researchers choose verbal data to address these questions, instead of other (e.g., reaction time, accuracy, multiple choice) types of data?
      • What are some differences between the two studies in terms of methods used (i.e., methods for collecting and analyzing verbal data), and how do those differences relate to the nature of the individual, talk-aloud data vs. collaborative, conversational data?
    • Optional: Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. Media:E&SPreface.pdf
  • 2-2-12 Protocol Analysis of Educational Discussions
    • Howley, I., Mayfield, E. & Rosé, C. P. (to appear). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc. Media:Chapter-Methods-Revised-Final.pdf
    • Discussion Questions:
      • What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
      • In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
      • Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
  • 2-7-12
    • van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7Media:VanSch7.pdf‎
    • Discussion Questions:
      • What, if any, of the steps described in this excerpt did you find confusing?
      • Which of these steps would you say are most methodologically challenging? most theoretically important?
      • How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?
    • [Dataset (please see zip file on blackboard)]
    • Example Coding Manuals Media:CopingEfficacy-v4.pdf Media:Negotiation_Addendum.pdf Media:Negotiation_Coding_Scheme.pdf
  • 2-9-12
    • Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[1]]
    • Discussion Questions:
      • What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
      • What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
      • What do you think is the most important caveat related to automatic analysis described in the paper?
  • 2-14-12
    • Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary] Media:ProtAna1.pdf
    • Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] Media:ProtAnalysis2.pdf
    • Discussion Questions:
      • What are the key features that make verbal protocols appropriate/not?
      • What can researchers do to collect and analyze such data most effectively?
    • Optional, supplementary reading: Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts Beyond Words: When Language Overshadows Insight, Journal of Experimental Psychology 122(2), pp 166-183. Media:Schooleretal.pdf‎
  • 2-16-10
    • Download SIDE and the SIDE User's Manual from the webpage. [[2]]
    • Discussion Questions:
      • What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
      • Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?
Cognitive Task Analysis - Revisited (Koedinger)
  • 2-21-12
    • Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences, 13 (2), 129-164.
    • Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. (pp. 471-476.) Austin, TX: Cognitive Science Society.
  • 2-23-12
    • Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.
  • Other optional readings on Difficulty Factors Assessment studies:
    • Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.
    • Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension. In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.
Psychometrics, reliability, Item Response Theory (Junker)
  • 2-28-12

1. From Trochim:

  A. Chapter 3 - the vocabulary of measurement 
  B. Chapter 5 - on constructing scales (it's ok to focus
      on the material up through sect 5.2a; the rest is
      more of a skim [but I'd be happy to talk about that 
      in class also])

2. On item response theory (IRT), a set of statistical models that are used to construct scales and to derive scores from them, especially in education and psychological research:

  A. Harris Article (PDF)
  Please take and self-score the test at the end of 
  this article.  Count each part of question one as
  one point, and each of the remaining three questions 
  as one point (no partial credit!).  Bring your 8
  scores to class.  E.g. if you missed 1(c) and (d), and
  you also missed question 4, then you would bring to
  class the following scores: 
  1 1 0 0 1 1 1 0
  If you missed 1(a) and (b) and question 2, bring the 
  following scores: 
  0 0 1 1 1 0 1 1 
  (note that the total score is 5 in both cases, but
  the pattern of rights and wrongs differs; it is the
  pattern that we are interested in).
  B. Please browse *online* through pp 1-23 of the pdf at
  The math is a bit heavy going but there are links 
  to apps that illustrate various points in the 
  harris article.  
  So skim the math and play with the apps.

We will discuss on Tue whatever of this we can get through, and continue the discussion as needed on Thursday. There will be additional readings for Thu to introduce fitting and using IRT models.

  • 3-1-12

1. Short introduction to R (Rintro.pdf)

   Please download and install R for your computer (windows, mac or
   linux) from [4].
   Then try all the things in sections 1-9 of this handout (section
   10 is optional).  You can try most things by copying from the pdf
   and pasting into the R command window.
   You do not have to be completely done with this by the time we
   meet for lecture 2, but you should aim to finish it soon

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma (Junker, Sijtsma (PDF))

   Please read up through p 266 only.
   The math is a bit heavy going so please try to read around it to
   see what the point of the article is.  
   We will try to look at some of the data in the article as examples
   in lecture 2.

3. "Psychometric Principles in Student Assessment" by Mislevy et al (Mislevy (PDF))

   Read through p 18.  This is a more modern modern look at some of
   the same issues that are addressed in Trochim's chapters.
   The remainder of this paper surveys various probabilistic models
   for the "measurement model" portion of Mislevy's framework (Figure
   1).  It is quite interesting but we will not pursue it.
  • 3-6-12 Psychometrics continued
Design Research & Qualitative Methods (Koedinger)
  • 3-8-12
    • Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
    • Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). PDF
    • Optional: Chapter on Design Research in Handbook of Learning Sciences
NO CLASS – Spring break 3-13-12 and 3-15-12
Surveys, Questionnaires, Interviews (Kiesler)
  • 3-20-12
    • Reading: Trochim Ch 4 and 5
  • 3-22-12
    • Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. Media:Tourangeau_SensitiveQuestions.pdf
    • Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. Media:Tourangeau_RememberingWhatHappened.pdf
Educational data mining (Scheines, Koedinger)
  • 3-27-12
    • Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. Media:RittterSchooler01.pdf
    • Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA Media:AIED2007_EDM_Zhang_ld_transfer.pdf
    • Assignment: Due Thursday, 4/1. DOC
  • 3-29-12
    • Register an account on DataShop ( ) and watch this video.
    • Reading: Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. PDF
  • 4-3-12
    • Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. Media:2000_roberts_pashler.pdf
    • Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. Media:GOF.doc No Summary Required.
  • 4-5-12
    • Do Unit 2 in the OLI course Empirical Research Methods
-- go to: 
-- go to Empirical Research methods (on left tab)
-- click on Peek In
-- complete Unit 2
    • Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. PDF
  • 4-10-12 Continuation of discussion of TETRAD
  • 4-12-12 Continuation of discussion of TETRAD
Experimental Research Methods (Koedinger)
  • 4-17-12
  • 4-19-12 NO CLASS – Spring Carnival
  • 4-24-12
    • Reading: Trochim Ch 9
    • Slides: pdf OR ppt
  • 4-26-12
    • Reading: Trochim Ch 10
    • Slides: pdf OR ppt
  • 5-1-12
    • Reading: Trochim Ch 14
    • Assignment 1 due before class
    • Try Regression and ANOVA modules of OLI Statistics course
  • 5-3-12 Course wrap-up