Theory Wiki - User contributions [en]

Educational Research Methods 2014

2014-01-14T03:15:54Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

===Research Methods for the Learning Sciences 05-748===
Spring 2014 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
5312 Wean Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and lit review due March 30, Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 14 (T)
* Choosing Qualitative & Quantitative Methods: Jan 16 (R)
* Video and Verbal Protocol Analysis: Jan 21, 23, 28, 30, Feb 4,6 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 11, 13, 18, 20 (TRTR)
* Educational Design Research: Feb 25 (R)
* Educational Measurement & Psychometrics: Feb 27, Mar 4, 6 (TRT)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 11, 13 (TR)
* Surveys, Questionnaires, Interviews: Mar 18, 20 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 25, 27, Apr 1 (TRT)
* Flex day: Apr 3 (R)
* Educational Data Mining & Causal Inference: Apr 8, 15, 17 (TTR)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 10 (R)
* Experimental Methods: Apr 22, 24, 29 (TRT)
* Wrap-up: May 1 (R)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-14
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]

*1-16 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 21]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 23 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 28 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Jan 30 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 4 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 6 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-11
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]

**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**[If you didn't have e-learning last semester] Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]

*1-23
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]*2-18
**Do one post on [[Media:Applying-CTA-assignment.docx|this assignment]] and a second post on the reading.
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. While the assignment is a rational CTA, note the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions in the assignment. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]

*2-20
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-25

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-27

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-4

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-6 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-11 and 3-13 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
* [Plans for these classes were communicated by Kiesler (& Koedinger) via email.]
*3-18
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-20
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-25
**Readings:
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2013_(Geom_Area_Unit_Spring_2010).doc | Learning-curve-assignment-2013.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. (See my emails, original and followup, for further directions on how to do this assignment.)
*3-27
**Read the following paper and make two posts on the general topic of this reading and the last, namely, using educational technology data as a basis for discovering improvements to cognitive models.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**Also, do some thinking about a semester project so we can discuss (and I can give feedback) on your possible ideas for a project.
*4-1
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***'''Optional:''' Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-3 To be used in case of rescheduling or for a student-driven topic.
***And/or for Review of Projects or Past Topics
**Option1. More on Educational Data Mining

**Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-8
**Before class on 4-8, do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2

*4-10 NO CLASS - Spring Carnival

*4-15
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-17 Continue discussion of Causal Inference from Data & TETRAD

=====Experimental Research Methods (Koedinger)=====
*4-22
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*4-24 NO CLASS
*4-29
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-1
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 9.

File:SpecialIssueAdamson-ThirdRevision Accepted.pdf

2014-01-14T03:13:10Z

Cprose:

Educational Research Methods 2014

2014-01-14T03:10:46Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

===Research Methods for the Learning Sciences 05-748===
Spring 2014 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
5312 Wean Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and lit review due March 30, Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 14 (T)
* Choosing Qualitative & Quantitative Methods: Jan 16 (R)
* Video and Verbal Protocol Analysis: Jan 21, 23, 28, 30, Feb 4,6 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 11, 13, 18, 20 (TRTR)
* Educational Design Research: Feb 25 (R)
* Educational Measurement & Psychometrics: Feb 27, Mar 4, 6 (TRT)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 11, 13 (TR)
* Surveys, Questionnaires, Interviews: Mar 18, 20 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 25, 27, Apr 1 (TRT)
* Flex day: Apr 3 (R)
* Educational Data Mining & Causal Inference: Apr 8, 15, 17 (TTR)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 10 (R)
* Experimental Methods: Apr 22, 24, 29 (TRT)
* Wrap-up: May 1 (R)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-14
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]

*1-16 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 21]: Connecting discussion and learning

**In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

**If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121.

**Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
***What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
***In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
***Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
***How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
***What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 23 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 28 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Jan 30 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 4 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 6 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-11
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]

**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**[If you didn't have e-learning last semester] Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]

*1-23
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]*2-18
**Do one post on [[Media:Applying-CTA-assignment.docx|this assignment]] and a second post on the reading.
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. While the assignment is a rational CTA, note the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions in the assignment. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]

*2-20
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-25

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-27

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-4

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-6 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-11 and 3-13 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
* [Plans for these classes were communicated by Kiesler (& Koedinger) via email.]
*3-18
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-20
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-25
**Readings:
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2013_(Geom_Area_Unit_Spring_2010).doc | Learning-curve-assignment-2013.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. (See my emails, original and followup, for further directions on how to do this assignment.)
*3-27
**Read the following paper and make two posts on the general topic of this reading and the last, namely, using educational technology data as a basis for discovering improvements to cognitive models.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**Also, do some thinking about a semester project so we can discuss (and I can give feedback) on your possible ideas for a project.
*4-1
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***'''Optional:''' Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-3 To be used in case of rescheduling or for a student-driven topic.
***And/or for Review of Projects or Past Topics
**Option1. More on Educational Data Mining

**Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-8
**Before class on 4-8, do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2

*4-10 NO CLASS - Spring Carnival

*4-15
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-17 Continue discussion of Causal Inference from Data & TETRAD

=====Experimental Research Methods (Koedinger)=====
*4-22
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*4-24 NO CLASS
*4-29
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-1
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 9.

File:PIERResearchMethodsPlan2014.doc

2014-01-14T03:02:31Z

Cprose:

Educational Research Methods 2013

2013-01-31T19:42:22Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

*Session 2[Jan 31 Carolyn]: Protocol Analysis of Collaborative Learning Discussions

**In this session we will explore the connection between talk and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse.

**Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J. & Rosé, C. (2012). Group Composition and Intelligent Dialogue Tutors for Impacting Students’ Academic Self-efficacy. Proceedings of the Intelligent Tutoring Systems Conference [[http://www.cs.cmu.edu/~emayfiel/application_papers/120113ITS12_ikh_07cpr.pdf]].

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Coding Manual for Negotiation [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Discussion Questions:
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in the Howley et al., 2012 study?

*Session 3[Feb 5 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 7 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 12 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 14 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://www.cs.cmu.edu/~emayfiel/side.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

File:Negotiation 10.pdf

2013-01-31T19:41:03Z

Cprose:

Educational Research Methods 2013

2013-01-31T19:40:06Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

*Session 2[Jan 31 Carolyn]: Protocol Analysis of Collaborative Learning Discussions

**In this session we will explore the connection between talk and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse.

**Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J. & Rosé, C. (2012). Group Composition and Intelligent Dialogue Tutors for Impacting Students’ Academic Self-efficacy. Proceedings of the Intelligent Tutoring Systems Conference [[http://www.cs.cmu.edu/~emayfiel/application_papers/120113ITS12_ikh_07cpr.pdf]].

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Coding Manual for Negotiation [[]]

*Discussion Questions:
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in the Howley et al., 2012 study?

*Session 3[Feb 5 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 7 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 12 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 14 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://www.cs.cmu.edu/~emayfiel/side.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

File:Qualitative-Quantitative-Chapt8.pdf

2013-01-28T13:51:37Z

Cprose:

Educational Research Methods 2013

2013-01-25T19:18:25Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

*Session 2[Jan 31 Carolyn]: Protocol Analysis of Collaborative Learning Discussions

**In this session we will explore the connection between talk and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse.

**Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J. & Rosé, C. (2012). Group Composition and Intelligent Dialogue Tutors for Impacting Students’ Academic Self-efficacy. Proceedings of the Intelligent Tutoring Systems Conference [[http://www.cs.cmu.edu/~emayfiel/application_papers/120113ITS12_ikh_07cpr.pdf]].

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

*Discussion Questions:
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in the Howley et al., 2012 study?

*Session 3[Feb 5 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 7 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 12 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 14 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://www.cs.cmu.edu/~emayfiel/side.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

File:VanSch7.pdf

2013-01-25T19:17:04Z

Cprose: uploaded a new version of "Image:VanSch7.pdf"

Educational Research Methods 2013

2013-01-25T18:15:13Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

*Session 2[Jan 31 Carolyn]: Protocol Analysis of Collaborative Learning Discussions

**In this session we will explore the connection between talk and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse.

**Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J. & Rosé, C. (2012). Group Composition and Intelligent Dialogue Tutors for Impacting Students’ Academic Self-efficacy. Proceedings of the Intelligent Tutoring Systems Conference [[http://www.cs.cmu.edu/~emayfiel/application_papers/120113ITS12_ikh_07cpr.pdf]].

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

*Discussion Questions:
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in the Howley et al., 2012 study?

*Session 3[Feb 5 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 7 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 12 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 14 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://www.cs.cmu.edu/~emayfiel/side.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

File:ProtAnalysis2.pdf

2013-01-25T18:14:01Z

Cprose: uploaded a new version of "Image:ProtAnalysis2.pdf"

File:ProtAna1.pdf

2013-01-25T18:12:31Z

Cprose: uploaded a new version of "Image:ProtAna1.pdf"

File:GihoolyEtAl2007.pdf

2013-01-25T18:09:04Z

Cprose:

Educational Research Methods 2013

2013-01-25T17:53:06Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

*Session 2[Jan 31 Carolyn]: Protocol Analysis of Collaborative Learning Discussions

**In this session we will explore the connection between talk and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse.

**Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J. & Rosé, C. (2012). Group Composition and Intelligent Dialogue Tutors for Impacting Students’ Academic Self-efficacy. Proceedings of the Intelligent Tutoring Systems Conference [[http://www.cs.cmu.edu/~emayfiel/application_papers/120113ITS12_ikh_07cpr.pdf]].

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

*Discussion Questions:
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in the Howley et al., 2012 study?

*Session 3[Feb 5 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625.

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 7 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning

*Discussion Questions:
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 12 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 14 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://www.cs.cmu.edu/~emayfiel/side.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

Educational Research Methods 2013

2013-01-25T17:51:29Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

*Session 2[Jan 31 Carolyn]: Protocol Analysis of Collaborative Learning Discussions

**In this session we will explore the connection between talk and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse.

**Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J. & Rosé, C. (2012). Group Composition and Intelligent Dialogue Tutors for Impacting Students’ Academic Self-efficacy. Proceedings of the Intelligent Tutoring Systems Conference.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

*Discussion Questions:
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in the Howley et al., 2012 study?

*Session 3[Feb 5 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625.

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 7 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning

*Discussion Questions:
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 12 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 14 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://www.cs.cmu.edu/~emayfiel/side.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

Educational Research Methods 2013

2013-01-25T17:49:53Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

*Session 2[Jan 31 Carolyn]: Protocol Analysis of Collaborative Learning Discussions

**In this session we will explore the connection between talk and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse.

**Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J. & Rosé, C. (2012). Group Composition and Intelligent Dialogue Tutors for Impacting Students’ Academic Self-efficacy. Proceedings of the Intelligent Tutoring Systems Conference.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[Image:Chapter-Methods-Revised-Final.pdf]]

*Discussion Questions:
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in the Howley et al., 2012 study?

*Session 3[Feb 5 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625.

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 7 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning

*Discussion Questions:
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 12 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 14 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://www.cs.cmu.edu/~emayfiel/side.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

Educational Research Methods 2013

2013-01-25T17:46:46Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

*Session 2[Jan 31 Carolyn]: Protocol Analysis of Collaborative Learning Discussions

**In this session we will explore the connection between talk and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse.

**Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J. & Rosé, C. (2012). Group Composition and Intelligent Dialogue Tutors for Impacting Students’ Academic Self-efficacy. Proceedings of the Intelligent Tutoring Systems Conference.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.

*Discussion Questions:
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in the Howley et al., 2012 study?

*Session 3[Feb 5 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625.

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 7 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning

*Discussion Questions:
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 12 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 14 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://www.cs.cmu.edu/~emayfiel/side.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

Educational Research Methods 2013

2013-01-25T17:41:39Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-22 and 1-24 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about what tasks you would use, what CTA technique(s), and how might represent the output of your analysis.
**Slides: [[Media:CTA2-2013.pdf|CTA2-2013.pdf]]
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) how do the authors represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
**In the first forum (where you posted one of your research topics), reply to your thread with a post that describes an example task that you could productively analyze in your domain of interest. You might also indicate some variations on the task that might help reveal what is most challenging for learners.
**Slides: [[Media:CTA3-2013.pdf|CTA3-2013.pdf]]
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

Educational Research Methods 2013

2013-01-21T05:36:19Z

Cprose: /* Class URLs */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-24 and 1-26 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about that domain, how you might perform a CTA, and you could represent the output of your analysis.
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to how the authors represent the output of their analysis: Do they use production rules? What kinds of diagrams do they use?
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

Educational Research Methods 2013

2013-01-21T05:34:11Z

Cprose: /* Class URLs */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

Summary Table: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE#gid=0]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA1-2013.pdf|CTA1-2013.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-24 and 1-26 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about that domain, how you might perform a CTA, and you could represent the output of your analysis.
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to how the authors represent the output of their analysis: Do they use production rules? What kinds of diagrams do they use?
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

Educational Research Methods 2013

2013-01-16T02:18:53Z

Cprose: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

==Research Methods for the Learning Sciences 05-748==
Spring 2013 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
3001 Newell Simon Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2013]

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

====Flipped Homework: Reading Reports and Pre-Class Assignments====

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well.
:# Think of it as writing a grant proposal. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. Since this is styled as a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

====Class Schedule in Brief====
* Course Intro: Formulating Good Research Questions: Jan 15 (T)
* Cognitive Task Analysis 1: Jan 17, 22, 24 (RTR)
* Video and Verbal Protocol Analysis: Jan 29, 31, Feb 5,7,12,14 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Cognitive Task Analysis 2: Feb 19, 21 (TR)
* Educational Measurement & Psychometrics: Feb 26, 28, Mar 5 (TRT)
** Guest Instructor: Brian Junker
* Educational Design Research: Mar 7 (R)
* NO CLASS – Spring break, Mar 12, 14 (TR)
* Surveys, Questionnaires, Interviews: Mar 19, 21 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 26, 28, Apr 2 (TRT)
* Flex day: Apr 4 (R)
* Educational Data Mining & Causal Inference: Apr 9, 11, 16 (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 18 (R)
* Experimental Methods: Apr 23, 25, 30 (TRT)
* Wrap-up: May 2 (R)

====Class Schedule with Readings and Assignments====

'''NOTE:''' This is a "living" document. It carries over some elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro & Formulating Good Research Questions (Koedinger)=====
*1-15
**See your email or [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard] for the pre-class assignment.
**[[Media:CourseIntroGoodQuestions13.pdf|Lecture slides]]
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter1]]
**[Optional (re)reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

=====Cognitive Task Analysis (Koedinger) =====
*1-17
**Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]
**Do a couple short assignments here: http://Assistment.org. Please create and an account, click on "Tutor", "Enroll in a class", select "Ken Koedinger" and "Educational Research Methods".
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]
**[Optional reading] Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
*1-22
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates. [[Media:Clarketal2007-CTAchapter.pdf|Clarketal2007-CTAchapter.pdf]]
***One point of reflection for you on the Clark et al reading is to compare and contrast the Cognitive Task Analysis (CTA) methods and output representations recommended with the approach taken by Zhu & Simon. Also, note their examples and claims about the power of CTA for improving instruction. (If you saw Bror Saxberg's recent PIER talk, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". How is this claim similar or different from Zhu & Simon? The notion of adaptive expertise is also important and interesting.
***As you read the 1-24 and 1-26 readings, be thinking about steps you could take to do a cognitive task analysis, empirical and rational, in a domain of your interest. Think about that domain, how you might perform a CTA, and you could represent the output of your analysis.
*1-24
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
**Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) outside of math domains. The Aleven et al reading provides an example of a CTA at the level of metacognitive skills. The Siegler reading shows a CTA dealing with younger kids. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. When you skim all three, pay particular attention to how the authors represent the output of their analysis: Do they use production rules? What kinds of diagrams do they use?
*Other possible readings:
**Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for these six sessions in 2013, 1-29 to 2-14, is in [[Media:PIERResearchMethodsPlan2013.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://www.cs.cmu.edu/~emayfiel/side.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 29]: Overview of Protocol Analysis

**In this introductory discussion, we will explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

=====Cognitive Task Analysis - Revisited (Koedinger) =====
*2-19
**Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
**Optional: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper provides a nice illustration. Think about how could you create a model of skills needed for these tasks and use it to predict or account for the key differences observed in the data? After having come up with some ideas, compare them with the optional reading, Koedinger & McLaren. Make at least one related post.
**Write another post briefly indicating how you might perform a CTA in a domain of your interest. Which kind(s) of CTA would you employ and why?
*2-21
**Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
**One thing that struck me about our conversations last time about applying CTA to your work is that it was sometimes difficult to track the connection to the research goal. Let's make that goal explicit here. Make one post that states (one of) your key research question(s) as related to learning research.
**Make another post about the Koedinger & McLaughlin reading. The main point of this reading is to provide an illustration of the translation of CTA into an instructional innovation and an evaluation of the innovation. In a second post, describe the strategy used to translate from CTA to instructional innovation. (If you are having trouble, try first to describe the key CTA outcome and what is the innovation.)
*Other optional readings
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
**Koedinger, K. R., Corbett, A. C., & Perfetti, C. (in press). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

NEW ASSIGNMENTS

*2-26

Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-5

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

=====Design Research & Qualitative Methods (Koedinger) =====
*3-7
**Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
**Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
**Optional: Chapter on Design Research in Handbook of Learning Sciences

=====NO CLASS – Spring break 3-12 and 3-14 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-19
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-21
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-26
**Readings:
***Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
**'''Short assignment:''' The assignment ([[Media:Learning-curve-assignment-2012.doc | Learning-curve-assignment-2012.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. [[Media:Geometry_-_Unit_Area.pdf | This file (Geometry_-_Unit_Area.pdf)]] will be needed to help answer questions Q6-Q7.
**On the discussion board, 1) post a comment or question about at least one of the two readings and 2) attach your assignment and/or bring a hard copy of it to class. That is, only one post is required (but more than one is welcome).
*3-28
**Readings:
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. D. Ashley, T.-W. Chan (Eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 164-175. Berlin: Springer-Verlag. [http://learnlab.org/uploads/mypslc/publications/learning_factor_analysis_5.2.pdf PDF]
***'''Optional:''' Martin, B., Mitrovic, T., Mathan, S., & Koedinger, K.R. (2011). Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction: The Journal of Personalization Research (UMUAI). 21(3), pp. 249-283. [[Media:xxx | file]]
*4-2
**Please finish off one of the two exercises you started for last class. See A or B further below. In either case, provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard.
**ALSO, make a post about your idea for a course final project. What method might you apply to address what research question?
**No required reading assignment.
***Optional readings:
***Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-4 To be used in case of rescheduling or for a student-driven topic.
**And/or for Review of Projects or Past Topics
***Make some progress on your course project and bring your ideas/questions to class. On the discussion board, please reply to my reply about your posted project idea to commit to an idea or elaborate on your plan.
***Regarding methods we have addressed, it seems there were some lingering questions at the end of the last couple of sessions related to statistical analysis and/or design research strategies. We can talk in class about those.

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-9
**Do Unit 2 in the OLI course Empirical Research Methods
go to: http://oli.web.cmu.edu/openlearning/
in the left tab, go to "Prior work..." and then "Empirical Research Methods"
click on Peek In
complete Unit 2
**Read Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]
*4-11 Continue discussion of Causal Inference from Data & TETRAD
*4-16 Continue discussion of TETRAD

*4-18 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-23
**Reading: Trochim Ch 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*4-25
**Reading: Trochim Ch 9
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*4-30
**Reading: Trochim Ch 10
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]
*5-2
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 10.

Educational Research Methods 10

2010-02-11T15:12:54Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
**[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_7.pdf]]
**[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_4.pdf]]
**[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Media:Martin-WhiteChapter_3.pdf]]
** [Optional] Michaels et al., 2007 paper on Accountable Talk [[Media:2007_Deliberative_Discourse.pdf‎|AccountableTalkPaper]]
*2-11-10
** van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7[[Media:VanSch7.pdf‎]]
**Kumar, R., Ai, H., and Rosé (submitted). Choosing Optimal Levels of Social Interaction – Towards creating Human-like Conversational Tutors, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010-thermo-rk.pdf|ITS2010-Kumar]]
** Data set from Kumar et al. study [[Media:Thermo-f09-transcripts.xls‎|Data]]
** Iris's coding manual [[Media:Thermo08CodingScheme-v9b.doc|Manual]]
** Chapter with alternative presentation of Reasoning coding [[Media:Sionti-Revised-Again.doc|Chapter]]
*2-16-10
** Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[http://www.cs.cmu.edu/~cprose/pubweb/Proof2.pdf]]
** Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010_submission_172.pdf]]‎
*2-18-10
** Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts Beyond Words: When Language Overshadows Insight, Journal of Experimental Psychology 122(2), pp 166-183. [[Media:Schooleretal.pdf‎]]
*2-23-10
** Download SIDE and the SIDE User's Manual from the webpage. [[http://www.cs.cmu.edu/~cprose/SIDE.html]]

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

File:2007 Deliberative Discourse.pdf

2010-02-11T15:11:29Z

Cprose: Overview of Accountable Talk work

Overview of Accountable Talk work

Educational Research Methods 10

2010-02-10T19:46:16Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
**[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_7.pdf]]
**[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_4.pdf]]
**[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Media:Martin-WhiteChapter_3.pdf]]
*2-11-10
** van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7[[Media:VanSch7.pdf‎]]
**Kumar, R., Ai, H., and Rosé (submitted). Choosing Optimal Levels of Social Interaction – Towards creating Human-like Conversational Tutors, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010-thermo-rk.pdf|ITS2010-Kumar]]
** Data set from Kumar et al. study [[Media:Thermo-f09-transcripts.xls‎|Data]]
** Iris's coding manual [[Media:Thermo08CodingScheme-v9b.doc|Manual]]
** Chapter with alternative presentation of Reasoning coding [[Media:Sionti-Revised-Again.doc|Chapter]]
*2-16-10
** Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[http://www.cs.cmu.edu/~cprose/pubweb/Proof2.pdf]]
** Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010_submission_172.pdf]]‎
*2-18-10
** Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts Beyond Words: When Language Overshadows Insight, Journal of Experimental Psychology 122(2), pp 166-183. [[Media:Schooleretal.pdf‎]]
*2-23-10
** Download SIDE and the SIDE User's Manual from the webpage. [[http://www.cs.cmu.edu/~cprose/SIDE.html]]

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

Educational Research Methods 10

2010-02-10T19:45:20Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
**[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_7.pdf]]
**[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_4.pdf]]
**[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Media:Martin-WhiteChapter_3.pdf]]
*2-11-10
** van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7[[Media:VanSch7.pdf‎]]
**Kumar, R., Ai, H., and Rosé (submitted). Choosing Optimal Levels of Social Interaction – Towards creating Human-like Conversational Tutors, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010-thermo-rk.pdf|ITS2010-Kumar]]
** Data set from Kumar et al. study [[Media:Thermo-f09-transcripts.xls‎|Data]]
** Iris's coding manual [[Media:Thermo08CodingScheme-v9b.doc|Manual]]
** Chapter with alternative presentation of Reasoning coding [[Media::Sionti-Revised-Again.doc|Chapter]]
*2-16-10
** Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[http://www.cs.cmu.edu/~cprose/pubweb/Proof2.pdf]]
** Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010_submission_172.pdf]]‎
*2-18-10
** Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts Beyond Words: When Language Overshadows Insight, Journal of Experimental Psychology 122(2), pp 166-183. [[Media:Schooleretal.pdf‎]]
*2-23-10
** Download SIDE and the SIDE User's Manual from the webpage. [[http://www.cs.cmu.edu/~cprose/SIDE.html]]

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

File:Thermo08CodingScheme-v9b.doc

2010-02-10T19:41:21Z

Cprose: Iris's coding manual

Iris's coding manual

File:Sionti-Revised-Again.doc

2010-02-10T19:40:06Z

Cprose: Another perspective on coding Reasoning moves.

Another perspective on coding Reasoning moves.

Educational Research Methods 10

2010-02-04T21:20:13Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
**[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_7.pdf]]
**[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_4.pdf]]
**[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Media:Martin-WhiteChapter_3.pdf]]
*2-11-10
** van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7
**Kumar, R., Ai, H., and Rosé (submitted). Choosing Optimal Levels of Social Interaction – Towards creating Human-like Conversational Tutors, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010-thermo-rk.pdf|ITS2010-Kumar]]
** Data set from Kumar et al. study [[Media:Thermo-f09-transcripts.xls‎|Data]]
*2-16-10
** Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[http://www.cs.cmu.edu/~cprose/pubweb/Proof2.pdf]]
** Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010_submission_172.pdf]]‎
*2-18-10
** Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts Beyond Words: When Language Overshadows Insight, Journal of Experimental Psychology 122(2), pp 166-183.
*2-23-10
** Download SIDE and the SIDE User's Manual from the webpage. [[http://www.cs.cmu.edu/~cprose/SIDE.html]]

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

File:Thermo-f09-transcripts.xls

2010-02-04T21:19:09Z

Cprose:

Educational Research Methods 10

2010-02-04T21:10:52Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
**[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_7.pdf]]
**[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_4.pdf]]
**[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Media:Martin-WhiteChapter_3.pdf]]
*2-11-10
** van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7
**Kumar, R., Ai, H., and Rosé (submitted). Choosing Optimal Levels of Social Interaction – Towards creating Human-like Conversational Tutors, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010-thermo-rk.pdf|ITS2010-Kumar]]
** Data Set to be posted soon
*2-16-10
** Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[http://www.cs.cmu.edu/~cprose/pubweb/Proof2.pdf]]
** Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010_submission_172.pdf]]‎
*2-18-10
** Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts Beyond Words: When Language Overshadows Insight, Journal of Experimental Psychology 122(2), pp 166-183.
*2-23-10
** Download SIDE and the SIDE User's Manual from the webpage. [[http://www.cs.cmu.edu/~cprose/SIDE.html]]

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

Educational Research Methods 10

2010-02-04T21:09:26Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
**[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_7.pdf]]
**[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_4.pdf]]
**[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Media:Martin-WhiteChapter_3.pdf]]
*2-11-10
** van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7
**Kumar, R., Ai, H., and Rosé (submitted). Choosing Optimal Levels of Social Interaction – Towards creating Human-like Conversational Tutors, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010-thermo-rk.pdf|ITS2010-Kumar]]
** Data Set to be posted soon
*2-16-10
** Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[http://www.cs.cmu.edu/~cprose/pubweb/Proof2.pdf]]
** Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010_submission_172.pdf]]‎
*2-18-10
** Download SIDE and the SIDE User's Manual from the webpage. [[http://www.cs.cmu.edu/~cprose/SIDE.html]]
*2-23-10

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

File:Its2010-thermo-rk.pdf

2010-02-04T21:05:01Z

Cprose:

Educational Research Methods 10

2010-02-04T20:54:38Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
**[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_7.pdf]]
**[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Media:PedagogyChapter_4.pdf]]
**[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Media:Martin-WhiteChapter_3.pdf]]
*2-11-10
*2-16-10
** Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (In Press). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, submitted to the International Journal of Computer Supported Collaborative Learning [[http://www.cs.cmu.edu/~cprose/pubweb/Proof2.pdf]]
** Ai, H., Kumar, R., Nagasunder, A., Rose, C. P. (submitted). Exploring the Effectiveness of Social Capabilities and Goal Alignment in Computer Supported Collaborative Learning, submitted to the Intelligent Tutoring Systems Conference[[Media:Its2010_submission_172.pdf]]‎
*2-18-10
*2-23-10

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

File:Its2010 submission 172.pdf

2010-02-04T20:51:34Z

Cprose:

Educational Research Methods 10

2010-02-04T20:45:18Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
**[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Image:PedagogyChapter_7.pdf]]
**[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Image:PedagogyChapter_4.pdf]]
**[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Image:Martin-WhiteChapter_3.pdf]]
*2-11-10
*2-16-10
*2-18-10
*2-23-10

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

Educational Research Methods 10

2010-02-04T20:42:55Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Image:PedagogyChapter7.pdf]]
[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum. [[Image:PedagogyChapter4.pdf]]
[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave. [[Image:Martin-WhiteChapter 3]]
*2-11-10
*2-16-10
*2-18-10
*2-23-10

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

File:PedagogyChapter 7.pdf

2010-02-04T20:40:24Z

Cprose:

File:PedagogyChapter 4.pdf

2010-02-04T20:40:04Z

Cprose:

File:Martin-WhiteChapter 3.pdf

2010-02-04T20:38:51Z

Cprose:

Educational Research Methods 10

2010-02-04T20:37:15Z

Cprose:

==Research Methods for the Learning Sciences 85-748==
Spring 2010 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
336B Baker Hall for the first day.

3501 Newell Simon Hall thereafter.

====Instructors====
Professor Ken Koedinger

Location: 3601 Newell-Simon Hall

Phone: 8-7667

Email: Koedinger@cmu.edu

Office hours by appointment

Dr. Philip I. Pavlik Jr.

Location: 300S Craig St, 224

Phone: 8-1618

Email: ppavlik@andrew.cmu.edu

Office hours by appointment

=====Teaching Assistant=====
Benjamin Shih

Location: GHC 8003

Phone: 8-6289

Email: shih@cmu.edu

Office hours by appointment

====Class URL====
[http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_10 learnlab.org/research/wiki/index.php/Educational_Research_Methods_10]

====Goals====
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including overview of methods, cognitive task analysis, qualitative methods, protocol and discourse analysis, and educational data mining and log analysis. A key goal is to help students think about and learn how to apply these methods to their own research programs.

====Course Prerequisites====
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

====Textbook and Readings====
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

Other readings will be assigned in class.

====Reading Reports====
We will be using Google Wave for course reading reports and discussions. Google Wave combines discussion boards, instant messengers, and wikis into a single system. You can use it just as you would a discussion board, but you can also edit your own / other peoples' posts, play back the changes, and see changes update in real-time. Further details and account invitations will be discussed in class.

Reading reports consist of three parts: students are required to submit at least one original post per reading assignment, at least one reply or comment on another student's post, and at least one substantive addition to the reading assignment summary. More posts, replies, and summary improvements are encouraged.

{| class="wikitable" border="1"
|-
! Monday
! Tuesday
! Wednesday
! Thursday
! Friday
! Saturday
! Sunday
|-
| Original Post (Tuesday Reading)
| Reply (Tuesday Reading)
Summary Edits (Tuesday Reading)
|
| Original Post (Thursday Reading)
Summary Edits (Thursday Reading)
|
|
| Reply (Thursday Reading)
|}

=====Posts and Replies=====

Original posts should contain at least one of the following:
*a question you had about the reading or something important you did not understand
*an idea inspired by the reading
*an interesting connection with something you learned or did previously in this or another course, or in other professional work or research

For readings due on a Tuesday, the original post must be submitted by Monday morning.

For readings due on a Thursday, the original post must be submitted by Thursday morning.

Replies must be:
*an on-topic, relevant response, clarification, or further comment on another student’s post

For readings due on a Tuesday, at least one reply must be submitted by Tuesday morning.

For readings due on a Thursday, at least one reply must be submitted by Sunday morning.

This means that replies for Tuesday readings are due before class, whereas replies for Thursday readings are due after. Please use this extra time to have a full and meaningful discussion on the topics discussed.

=====Summary=====

For each reading assignment, one student will be responsible for a finished summary of that assignment and its related discussion.

Each summary will consist of:
* A brief overview of the reading assignment. For a chapter from the textbook, this should be a couple sentences on major topics addressed in the chapter. For a research paper, this should be a couple sentences covering the research question(s) and primary result(s).
* A brief discussion of the methodology. For a chapter from the textbook, this should be a more detailed discussion of the main research methodology discussed. For a research paper, this should be a couple sentences discussing aspects of the data, such as the subject population or analytical methods.
* A listing of major issues or suggestions for the paper, as related to the course. Threats to validity and problems with test reliability are example topics, as well as suggestions on how to avoid or resolve such issues.

The first two parts of the summary should be complete by the morning of the day of class.

====Grading====

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by your participation in Reading Reports, and by your participation in class.

* Course work
** 10% Reading reports
** 50% Homework assignments
* Project & final paper
** 40% Design a new study based on one (or more) of these methods that pushes your own research in a new direction.

====Class Schedule====

=====Basic Research & Experimental Methods (Koedinger, Pavlik)=====
*1-12-10
**Slides: [[Media:L01-CourseIntroGoodQuestions.pdf|pdf version]] OR [[Media:L01-CourseIntroGoodQuestions.ppt|ppt version]]
**Assignment 1: [[Media:Assignment1.doc|Assignment1.doc]]
***Assignment 1 Paper: [[Media:1996_Marcus.pdf|1996_Marcus.pdf]]
*1-14-10
**Reading: Trochim Ch 1 and 7
**Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|ppt]]
*1-19-10
**Reading: Trochim Ch 9 and 10
**Slides: [[Media:L03-True-Experiments.pdf|pdf]] OR [[Media:L03-True-Experiments.ppt|ppt]]
*1-21-10
**Reading: Trochim Ch 11
**Assignment 1 due before class
**Slides: [[Media:L04-quasi-experiments.pdf|pdf]] OR [[Media:L04-quasi-experiments.ppt|ppt]]

=====Cognitive Task Analysis (Koedinger, Pavlik) =====
*1-26-10
**Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2007). Cognitive task analysis: In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). Mahwah, NJ: Lawrence Erlbaum Associates.
**Assignment 2: [[Media:Assignment2.doc|Assignment2.doc]]
**Slides: [[Media:CTA-01.pdf|CTA-01.pdf]]

*1-28-10
**Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.

**Heffernan, N. & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension: In Shafto, M. G. & Langley, P. (Eds.) Proceedings of the Nineteenth Annual Conference of the Cognitive Science Society, (pp. 307-312). Hillsdale, NJ: Erlbaum.

*2-2-10
**Go to PSLC poster session outside of 6115 Gates
**See projects from these pages [[Cognitive Factors]], [[Metacognition and Motivation]], [[Social and Communicative Factors in Learning]], and [[Computational Modeling and Data Mining]]
**Reading for later: How People Learn Chapter 2: How Experts Differ From Novices

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
*2-4-10: In this introductory lecture, we will discuss the main steps of protocol analysis and what can be gained from the process. We will discuss these 2 readings in class.
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal Reports as Data (Revised Edition, pp. xii-xv). Cambridge, MA: MIT Press. [[Media:E&SPreface.pdf]]
**Gilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, 611-625. [[Media:Gilhooly.pdf‎]]
*2-9-10 Protocol Analysis of Educational Discussions
[Half the class will read this one]Veel, R. (1999). Language, knowledge and authority in school mathematics, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum.
[Half the class will read this one] Williams, G. (1999). The pedagogic device and the production of pedagogic discourse: a case example in early literacy education, in Francis Christie (Ed.) Pedagogy and the Shaping of Consciousness: Linguistics and Social Processes, Continuum.
[Optional] Martin, J. and White, P. R. (2005). The Language of Evaluation: Appraisal in English, Chapter 3, Palgrave.
*2-11-10
*2-16-10
*2-18-10
*2-23-10

=====Psychometrics, reliability, Item Response Theory (Junker, Koedinger)=====
*2-25-10
*3-2-10
*3-4-10
=====NO CLASS – Spring break=====
*3-9-10
*3-11-10
=====OPEN? [Design Research?] =====
*3-16-10
=====Surveys, Questionnaires, Interviews (Kiesler) =====
*3-18-10
*3-23-10
=====Educational data mining (Scheines, Pavlik, Koedinger) =====
*3-25-10
*3-30-10
*4-1-10
*4-6-10
*4-8-10
*4-13-10
*4-15-10 NO CLASS – Spring Carnival
=====Cognitive Task Analysis - Revisited (Koedinger, Pavlik) =====
*4-20-10
*4-22-10
=====Wrap-up=====
*4-27-10
*4-29-10

Walker - Features of Adaptive Assistance that Improve Peer Tutoring in Algebra

2010-01-31T04:29:36Z

Cprose: New page: In our project, we analyze how conceptual and elaborated content in student dialogue during a peer tutoring activity influences the learning of both the help-giver and the help-receiver. W...

In our project, we analyze how conceptual and elaborated content in student dialogue during a peer tutoring activity influences the learning of both the help-giver and the help-receiver. We further investigate how we can support poor collaborators in producing those beneficial interactions by automatically assessing student dialogue and providing assistance when appropriate. We have recently conducted a study comparing adaptive support to peer tutoring to fixed support, and found that the adaptive support indeed improved student peer tutoring interactions. We intend to take two further steps: 1) Continue analyzing the data to examine whether the improved interaction related to increased collaboration skills and learning outcomes (to be completed by October 15), and 2) Conduct a user study examining how public and private features of different types of assistance relate to student feelings of accountability and subsequent likelihood to incorporate the assistance into their interactions (to be completed by January 15). By the completion of the project, which we plan for at the end of May, we will have an increased understanding of how specific features of adaptive assistance can support students in interacting productively and thus in learning from collaboration. By that time, we will have analyzed the data from the January study, and we will have produced an adaptive system capable of supporting peer tutoring in Algebra, and demonstrated how features of the adaptive system influence student peer tutoring behavior.

This project relates the more general thrust goals as follows. It is examining how features of assistance affect the three aspects of accountable talk: accountability to knowledge, accountability to rigorous thinking, and accountability to the learning community. Steps are being made toward the ambitious goal to operationalize and assess these aspects of accountable in real time as students interact and receive assistance in this computer-mediated environment. There is also a potential to code the three way dialog (student tutee, student tutor, and computer tutor) for transactivity. In particular, the student tutee's dialog moves have not yet been coded, but appear to have interesting elements, like asking for specific help or self-explaining, that may well connect to transactivity codes. Finally, there is a potential to analyze the computer tutor's reflective prompts for similarity with accountable talk moves and associated effectiveness. Some of the prompts were indeed inspired by accountable talk moves.

Resnick Project

2010-01-31T04:27:46Z

Cprose: New page: In the Social-Communicative Factors thrust, we are investigating communication as a core enabler of robust learning, including detailed study of patterns of verbal interaction, the role of...

In the Social-Communicative Factors thrust, we are investigating communication as a core enabler of robust learning, including detailed study of patterns of verbal interaction, the role of social variables in initiating and sustaining learning, and the effects on motivation, self-attribution and commitment to a learning group that are associated with learning through social-communicative interaction. In several prior studies we have seen remarkable effects of active classroom discussion where students articulate their reasoning and build on the reasoning of their fellow students in comparison with students in control classrooms where they have not had the benefit of this teacher orchestrated interaction with their peers. These effects have been measured on standardized tests in math in comparison with control classrooms. We are currently working to set up a LearnLab where we can study Accountable Talk in the PSLC LearnLab style.

Rose - Integrated framework for analysis of classroom discussions

2010-01-31T04:25:20Z

Cprose:

Within the field of computer supported collaborative learning, the topic of what makes group discussions productive for learning and community building has been explored with very similar findings, perhaps with subtle distinctions, and under different names such as transactivity (Berkowitz & Gibbs, 1983; Teasley, 1997; Weinberger & Fischer, 2006) in the cognitive learning community and uptake (Suthers, 2006), group cognition (Stahl, 2006), or productive agency (Schwartz, 1998) in the socio-cultural learning community. Despite differences in orientation between the cognitive and socio-cultural learning communities, the conversational behaviors that have been identified as valuable are very similar. And building on these common findings, the field of Computer-Supported Collaborative Learning has emerged where support for collaborative learning has been developed that addresses observed weaknesses in conversational behavior related to this phenomenon.

In order to deepen and expand our understanding of what has been called ‘transactivity’ in the literature on collaborative dyadic interaction, we are attempting to extend those ideas to student discourse in the context of classroom discussion. The de Lisi and Golbeck interpretation of Piaget’s theory models the process through which experiences with peers can play a critical role in the development of a child’s cognitive system. A key idea that has been appropriated from this theory is that when students come together to solve a problem, bringing with them different perspectives, the interaction causes the participants to consider questions that might not have occurred to them otherwise. Through this interaction, children operate on each other’s reasoning, and in so doing they become aware of inconsistencies between their reasoning and that of their partner or even within their own model itself (Teasley). This process was termed transactive discussion after Dewey and Bentley(1949), and further formalized by Berkowitz and Gibbs (1980, 1983, manual). A transactive discussion is defined most simply as “reasoning that operates the reasoning of another” (Berkowitz and Gibbs 1983), although the Berkowitz and Gibbs formulation also allows for transactive contributions to operate on formerly expressed reasoning of the speaker himself.

Explicitly articulated reasoning and transactive discussion are at the heart of what makes collaborative learning discussions valuable. When we shift to consider teacher-guided classroom discourse we will still find similar collaborative exchanges between peers, but there it will be enriched with the pedagogical lead of the teacher. The teacher is responsible for orchestrating the discussion and setting up a structure that is used to elicit reasoned participation from the students.

Any transcript can be coded in limitless ways. Our choice of code is driven by certain hypotheses about what kinds of peer to peer or teacher and student discourse will promote robust learning. We are seeking to make those as precise as possible, so that we can operationalize the discourse categories into a codable form and study them systematically. However, in classroom situations, where the teacher plays the role of lead orchestrator of talk, there is the need to code teacher and student discourse differently, in order to develop quantifiable measures of the kinds of teaching and classroom interaction we think are productive. That way we can test hypotheses of a variety of kinds whether a certain sequence of teacher moves frequently lead to a certain kind of student talk or if the quantity of a particular kind of student talk is associated with better learning outcomes (e.g. pre- post-test gains). We are not looking for the same thing in both teacher and student discourse, thus we do not code each utterance for the same things, whether the speaker is teacher or student.

If transactivity is defined as using another person’s thinking actively to change your own thinking, or develop your own thinking, then the teacher is acting (in some classrooms) as a super proxy creating the conditions for everybody to experience transactivity vicariously or directly. We talk about this as scaffolded transactivity. Our team, including our colleagues at Boston University, are working to develop two complementary coding schemes, one that tracks student talk (lead by the CMU team), and one that tracks teacher moves that scaffold transactivity development in student talk (lead by the BU team).

Recently we have reviewed a large amount of literature from the area of systemic functional linguistics. Several relevant lines of work come from this community. First, they have a long track record for work on analysis of social interactions within traditional academic writing (Halliday, Martin, Rose, Christie, Hyland). In particular their Appraisal system includes sub-systems for characterizing how people position themselves through language in relation to their listeners, the content they are communicating, and the relationship between that content and that of previous contributions (e.g., earlier publications). Most relevant is their work on a sub-system they refer to as Engagement. Some of this work has already been adapted to face-to-face conversation, including whole group class discussions about math, where in addition to an analysis of these interpersonal/relational aspects of language, an analysis of linguistic constructions that are useful for articulating math concepts and which can be use as an indicator for the level of sophistication in a student’s math articulation, which may be useful for our work from an assessment standpoint. Part of their goal has been to make the act of positioning, which Martin and Rose characterize as “power relationships” within the texts, explicit in order to make those positioning processes explicit and teachable. Another aspect of their work that builds on this is work on literacy issues, especially for traditionally “low power” populations, such as aboriginal communities in Australia.

A major aspect of the work we are doing involves automatic analysis of discussion data. Some publically available tools we have produced are found at http://www.cs.cmu.edu/~cprose/TagHelper.html and http://www.cs.cmu.edu/~cprose/SIDE.html.

Machine-learning algorithms can learn mappings between a set of input features and a set of output categories. They do this by using statistical techniques to find characteristics of hand-coded “training examples” that exemplify each of the output categories. The goal of the algorithm is to learn rules by generalizing from these examples in such a way that the rules can be applied effectively to new examples. In order for this to work well, the set of input features provided must be sufficiently expressive, and the training examples must be representative. Typically, machine-learning researchers design a set of input features that they suspect will be expressive enough. At the most superficial level, these input features are simply the words in a document. But many other features are routinely used in a wide range of text-processing applications, such as word collocations and simple patterns involving part of speech tags and low-level lexical features; we will draw from this prior work.

Once candidate input features have been identified, analysts typically hand code a large number of training examples. The previously developed TagHelper tool set (Rosé et al., 2008) has the capability of allowing users to define how texts will be represented and processed by making selections on the GUI interface. In addition to basic text-processing tools such as part-of-speech taggers and stemmers that are used to construct a representation of the text that machine-learning algorithms can work with, a variety of algorithms from toolkits such as Weka (Witten & Frank, 2005) are included in order to provide many alternative machine-learning algorithms to map between the input features and the output categories. Based on their understanding of the classification problem, machine-learning practitioners typically pick an algorithm that they expect to perform well. Often this is an iterative process of applying an algorithm, seeing where the trained classifier makes mistakes, and then adding additional input features, removing extraneous input features, or experimenting with algorithms.

Applying this iterative process requires insight and skill in the areas of linguistics and machine learning that the social scientists conducting corpus analysis are unlikely to possess. TagHelper tools supports this interactive processes by making it easy to define different processing configurations through the GUI and then providing reports about how the configuration worked and where the process may have broken down. The goal of our tool development is to make this process easier for social scientists. In particular, the process of identifying where the process has broken down and how the configuration can be tuned in order to improve the performance requires more expertise than typical social scientists would possess. Thus, the bulk of our development work will be in developing the machinery to bridge the gap between the natural structure of the input texts and the behaviors that social scientists are interested in cataloguing and coding, using bootstrapping approaches.

In our recent corpus-based experiments (Josh & Rosé, 2009; Arora, Joshi, & Rosé, 2009) we have explored the usage of alternative types of syntactically motivated features on text classification performance. Our methodology is extensively discussed in our recent journal article in the International Journal of Computer-Supported Collaborative Learning, investigating the use of text classification technology for automatic collaborative learning process analysis (Rosé et al., 2008). In more recent work we have experimented with learning paradigms such as genetic programming and genetic algorithms to “evolve” more powerful features that improve classification performance.

Rose - Integrated framework for analysis of classroom discussions

2010-01-31T04:24:05Z

Cprose: New page: Within the field of computer supported collaborative learning, the topic of what makes group discussions productive for learning and community building has been explored with very similar ...

Within the field of computer supported collaborative learning, the topic of what makes group discussions productive for learning and community building has been explored with very similar findings, perhaps with subtle distinctions, and under different names such as transactivity (Berkowitz & Gibbs, 1983; Teasley, 1997; Weinberger & Fischer, 2006) in the cognitive learning community and uptake (Suthers, 2006), group cognition (Stahl, 2006), or productive agency (Schwartz, 1998) in the socio-cultural learning community. Despite differences in orientation between the cognitive and socio-cultural learning communities, the conversational behaviors that have been identified as valuable are very similar. And building on these common findings, the field of Computer-Supported Collaborative Learning has emerged where support for collaborative learning has been developed that addresses observed weaknesses in conversational behavior related to this phenomenon.

In order to deepen and expand our understanding of what has been called ‘transactivity’ in the literature on collaborative dyadic interaction, we are attempting to extend those ideas to student discourse in the context of classroom discussion. The de Lisi and Golbeck interpretation of Piaget’s theory models the process through which experiences with peers can play a critical role in the development of a child’s cognitive system. A key idea that has been appropriated from this theory is that when students come together to solve a problem, bringing with them different perspectives, the interaction causes the participants to consider questions that might not have occurred to them otherwise. Through this interaction, children operate on each other’s reasoning, and in so doing they become aware of inconsistencies between their reasoning and that of their partner or even within their own model itself (Teasley). This process was termed transactive discussion after Dewey and Bentley(1949), and further formalized by Berkowitz and Gibbs (1980, 1983, manual). A transactive discussion is defined most simply as “reasoning that operates the reasoning of another” (Berkowitz and Gibbs 1983), although the Berkowitz and Gibbs formulation also allows for transactive contributions to operate on formerly expressed reasoning of the speaker himself.

Explicitly articulated reasoning and transactive discussion are at the heart of what makes collaborative learning discussions valuable. When we shift to consider teacher-guided classroom discourse we will still find similar collaborative exchanges between peers, but there it will be enriched with the pedagogical lead of the teacher. The teacher is responsible for orchestrating the discussion and setting up a structure that is used to elicit reasoned participation from the students.

Any transcript can be coded in limitless ways. Our choice of code is driven by certain hypotheses about what kinds of peer to peer or teacher and student discourse will promote robust learning. We are seeking to make those as precise as possible, so that we can operationalize the discourse categories into a codable form and study them systematically. However, in classroom situations, where the teacher plays the role of lead orchestrator of talk, there is the need to code teacher and student discourse differently, in order to develop quantifiable measures of the kinds of teaching and classroom interaction we think are productive. That way we can test hypotheses of a variety of kinds whether a certain sequence of teacher moves frequently lead to a certain kind of student talk or if the quantity of a particular kind of student talk is associated with better learning outcomes (e.g. pre- post-test gains). We are not looking for the same thing in both teacher and student discourse, thus we do not code each utterance for the same things, whether the speaker is teacher or student.

If transactivity is defined as using another person’s thinking actively to change your own thinking, or develop your own thinking, then the teacher is acting (in some classrooms) as a super proxy creating the conditions for everybody to experience transactivity vicariously or directly. We talk about this as scaffolded transactivity. Our team, including our colleagues at Boston University, are working to develop two complementary coding schemes, one that tracks student talk (lead by the CMU team), and one that tracks teacher moves that scaffold transactivity development in student talk (lead by the BU team).

Recently we have reviewed a large amount of literature from the area of systemic functional linguistics. Several relevant lines of work come from this community. First, they have a long track record for work on analysis of social interactions within traditional academic writing (Halliday, Martin, Rose, Christie, Hyland). In particular their Appraisal system includes sub-systems for characterizing how people position themselves through language in relation to their listeners, the content they are communicating, and the relationship between that content and that of previous contributions (e.g., earlier publications). Most relevant is their work on a sub-system they refer to as Engagement. Some of this work has already been adapted to face-to-face conversation, including whole group class discussions about math, where in addition to an analysis of these interpersonal/relational aspects of language, an analysis of linguistic constructions that are useful for articulating math concepts and which can be use as an indicator for the level of sophistication in a student’s math articulation, which may be useful for our work from an assessment standpoint. Part of their goal has been to make the act of positioning, which Martin and Rose characterize as “power relationships” within the texts, explicit in order to make those positioning processes explicit and teachable. Another aspect of their work that builds on this is work on literacy issues, especially for traditionally “low power” populations, such as aboriginal communities in Australia.

A major aspect of the work we are doing involves automatic analysis of discussion data. Some publically available tools we have produced are found at the <a href="http://www.cs.cmu.edu/~cprose/TagHelper.html"> TagHelper page </a> and the <a href="http://www.cs.cmu.edu/~cprose/SIDE.html"> SIDE </a> page.

Machine-learning algorithms can learn mappings between a set of input features and a set of output categories. They do this by using statistical techniques to find characteristics of hand-coded “training examples” that exemplify each of the output categories. The goal of the algorithm is to learn rules by generalizing from these examples in such a way that the rules can be applied effectively to new examples. In order for this to work well, the set of input features provided must be sufficiently expressive, and the training examples must be representative. Typically, machine-learning researchers design a set of input features that they suspect will be expressive enough. At the most superficial level, these input features are simply the words in a document. But many other features are routinely used in a wide range of text-processing applications, such as word collocations and simple patterns involving part of speech tags and low-level lexical features; we will draw from this prior work.

Once candidate input features have been identified, analysts typically hand code a large number of training examples. The previously developed TagHelper tool set (Rosé et al., 2008) has the capability of allowing users to define how texts will be represented and processed by making selections on the GUI interface. In addition to basic text-processing tools such as part-of-speech taggers and stemmers that are used to construct a representation of the text that machine-learning algorithms can work with, a variety of algorithms from toolkits such as Weka (Witten & Frank, 2005) are included in order to provide many alternative machine-learning algorithms to map between the input features and the output categories. Based on their understanding of the classification problem, machine-learning practitioners typically pick an algorithm that they expect to perform well. Often this is an iterative process of applying an algorithm, seeing where the trained classifier makes mistakes, and then adding additional input features, removing extraneous input features, or experimenting with algorithms.

Applying this iterative process requires insight and skill in the areas of linguistics and machine learning that the social scientists conducting corpus analysis are unlikely to possess. TagHelper tools supports this interactive processes by making it easy to define different processing configurations through the GUI and then providing reports about how the configuration worked and where the process may have broken down. The goal of our tool development is to make this process easier for social scientists. In particular, the process of identifying where the process has broken down and how the configuration can be tuned in order to improve the performance requires more expertise than typical social scientists would possess. Thus, the bulk of our development work will be in developing the machinery to bridge the gap between the natural structure of the input texts and the behaviors that social scientists are interested in cataloguing and coding, using bootstrapping approaches.

In our recent corpus-based experiments (Josh & Rosé, 2009; Arora, Joshi, & Rosé, 2009) we have explored the usage of alternative types of syntactically motivated features on text classification performance. Our methodology is extensively discussed in our recent journal article in the International Journal of Computer-Supported Collaborative Learning, investigating the use of text classification technology for automatic collaborative learning process analysis (Rosé et al., 2008). In more recent work we have experimented with learning paradigms such as genetic programming and genetic algorithms to “evolve” more powerful features that improve classification performance.