Difference between revisions of "Educational Research Methods 2017"
(→Course Intro, Research Questions, Picking Methods (Koedinger)) |
(→Experimental Research Methods (Koedinger)) |
||
(28 intermediate revisions by 2 users not shown) | |||
Line 6: | Line 6: | ||
====Location==== | ====Location==== | ||
− | + | 4101 Gates/Hillman | |
====Instructor==== | ====Instructor==== | ||
Professor Ken Koedinger | Professor Ken Koedinger | ||
− | |||
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667 | Office: 3601 Newell-Simon Hall, Phone: 412-268-7667 | ||
+ | Email: Koedinger@cmu.edu, Office hours by appointment | ||
− | + | Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines | |
====Class URLs==== | ====Class URLs==== | ||
Line 28: | Line 28: | ||
===Textbook and Readings=== | ===Textbook and Readings=== | ||
− | "The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. | + | "The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. |
+ | |||
+ | Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here]. | ||
− | |||
− | |||
Other readings will be assigned in class. See below. | Other readings will be assigned in class. See below. | ||
Line 41: | Line 41: | ||
Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose. | Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose. | ||
− | Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before | + | Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well. |
These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class! | These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class! | ||
Line 74: | Line 74: | ||
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR) | * Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR) | ||
** Guest Instructors: Marsha Lovett & Carolyn Rose | ** Guest Instructors: Marsha Lovett & Carolyn Rose | ||
− | * Performing Cognitive Task Analysis: Feb 14, 16, 21 | + | * Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT) |
− | + | * Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT) | |
− | * Educational Measurement & Psychometrics: Mar 2, 7 | + | ** Guest Instructor: Rebecca Nugent |
− | ** Guest Instructor: | + | * Cognitive Task Analysis - Quantitative: Mar 9 (T) |
* NO CLASS – Spring break, Mar 14, 16 (TR) | * NO CLASS – Spring break, Mar 14, 16 (TR) | ||
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR) | * Surveys, Questionnaires, Interviews: Mar 21, 23 (TR) | ||
− | ** Guest Instructor: | + | ** Guest Instructor: Amy Ogan |
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT) | * Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT) | ||
− | * Flex day: Apr 6 (R) | + | * Flex day (Educational Design Research?): Apr 6 (R) |
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT) | * Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT) | ||
** Guest Instructor: Richard Scheines | ** Guest Instructor: Richard Scheines | ||
* NO CLASS – Spring Carnival, Apr 20 (R) | * NO CLASS – Spring Carnival, Apr 20 (R) | ||
− | * Experimental Methods: Apr 25, 27, May 2 ( | + | * Experimental Methods: Apr 25, 27, May 2, 4 (TRTR) |
− | * Wrap-up: May | + | * Wrap-up: May 9 (T) |
===Class Schedule with Readings and Assignments=== | ===Class Schedule with Readings and Assignments=== | ||
Line 96: | Line 96: | ||
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]] | **Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]] | ||
**Do the chpt 1 quiz | **Do the chpt 1 quiz | ||
− | **[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides | + | **[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]] |
*1-19 Choosing Qualitative & Quantitative Methods | *1-19 Choosing Qualitative & Quantitative Methods | ||
Line 108: | Line 108: | ||
=====Video and Verbal Protocol Analysis (Lovett, Rosé) ===== | =====Video and Verbal Protocol Analysis (Lovett, Rosé) ===== | ||
− | The | + | The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard. |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
=====Cognitive Task Analysis (CTA) (Koedinger) ===== | =====Cognitive Task Analysis (CTA) (Koedinger) ===== | ||
− | *2- | + | *2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts |
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]] | **[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]] | ||
**Do two posts on Blackboard. | **Do two posts on Blackboard. | ||
Line 211: | Line 119: | ||
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]] | ***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]] | ||
− | *2- | + | *2-16 Rational CTA via Cognitive Modeling |
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]] | **Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]] | ||
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]] | **[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]] | ||
Line 217: | Line 125: | ||
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]] | **[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]] | ||
− | *2- | + | *2-21 Doing CTA for higher-level thinking/learning skills |
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]] | **Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]] | ||
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]] | **Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]] | ||
Line 227: | Line 135: | ||
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]] | **A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]] | ||
− | * | + | *3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment |
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]] | **Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]] | ||
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis. | ***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis. | ||
Line 239: | Line 147: | ||
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]] | ***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]] | ||
− | =====Psychometrics, | + | =====Psychometrics, Reliability, Item Response Theory (Nugent)===== |
− | * | + | * TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent. |
− | *2- | + | *2-23 |
**Quick introduction to the R statistical language | **Quick introduction to the R statistical language | ||
Line 253: | Line 161: | ||
*** thermo11_data_integrated.csv - a data set for the examples. | *** thermo11_data_integrated.csv - a data set for the examples. | ||
− | *2- | + | *2-28 |
1. From Trochim: | 1. From Trochim: | ||
Line 298: | Line 206: | ||
So skim the math and play with the apps. | So skim the math and play with the apps. | ||
− | *3- | + | *3-2 |
The assignment for this lecture has two parts. | The assignment for this lecture has two parts. | ||
Line 326: | Line 234: | ||
in lecture 2. | in lecture 2. | ||
− | *3- | + | *3-7 Continued discussion of Psychometrics |
− | =====NO CLASS – Spring break 3- | + | =====NO CLASS – Spring break 3-14 and 3-16 ===== |
− | =====Surveys, Questionnaires, Interviews ( | + | =====Surveys, Questionnaires, Interviews (Ogan) ===== |
− | + | *3-21 | |
− | *3- | ||
**Reading: Trochim Ch 4 and 5 | **Reading: Trochim Ch 4 and 5 | ||
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question. | ***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question. | ||
− | *3- | + | *3-23 |
− | **Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. | + | **Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes. |
**Readings | **Readings | ||
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]] | ***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]] | ||
Line 342: | Line 249: | ||
=====Educational Data Mining -- Learning Curve Analysis (Koedinger) ===== | =====Educational Data Mining -- Learning Curve Analysis (Koedinger) ===== | ||
− | *3- | + | *3-28 |
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS | **BRING YOUR LAPTOP FOR ALL THESE SESSIONS | ||
− | **Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday | + | **Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday). |
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data. | ***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data. | ||
**Read the following paper and make two posts as usual. | **Read the following paper and make two posts as usual. | ||
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]] | ***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]] | ||
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]] | ***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]] | ||
− | **'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) | + | **'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) comfortably before class on Thursday -- by 3pm. Also, in addition to the problem content file indicated in the assignment handout see other files in the same location to get a more complete description and list of the files: Geometry Area Problems PDF Explanation.docx and solutions.zip. |
− | *3- | + | *3-30 |
**Read the following paper and make two posts as usual. | **Read the following paper and make two posts as usual. | ||
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]] | ***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]] | ||
− | **In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4- | + | **In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (at least get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-5. |
− | *4- | + | *4-4 |
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B). | **In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B). | ||
**Read the following paper and make two posts as usual. | **Read the following paper and make two posts as usual. | ||
Line 362: | Line 269: | ||
Do A or B: | Do A or B: | ||
− | A. Modify a KC model in a DataShop dataset | + | A. Modify a KC model in a DataShop dataset |
− | 1. What is the DataShop dataset you modified? | + | 1. What is the DataShop dataset you modified? (Look for datasets with the lego block icon on them -- these have associated problem descriptions) |
2. Describe how you used the HMST procedure (from Stamper paper) | 2. Describe how you used the HMST procedure (from Stamper paper) | ||
to identify a KC to try to improve | to identify a KC to try to improve | ||
Line 369: | Line 276: | ||
KC file) & describe why you made the change you did | KC file) & describe why you made the change you did | ||
4. After importing your new KC model to DataShop, did it improve the | 4. After importing your new KC model to DataShop, did it improve the | ||
− | predictions | + | predictions on any of the metrics, AIC, BIC, or cross validation? |
(Caution: Make sure your new KC model labels the same number of | (Caution: Make sure your new KC model labels the same number of | ||
observations as the KC model you are modifying.) | observations as the KC model you are modifying.) | ||
B. Use R to create an alternative statistical model to AFM | B. Use R to create an alternative statistical model to AFM | ||
− | 1. Approximate | + | 1. Approximate AFM in R using either glm or glmer (in package lme4). You |
+ | can find R code that mimics AFM in the DataShop help, here: | ||
+ | https://pslcdatashop.web.cmu.edu/help?page=rSoftware | ||
+ | |||
+ | How do the parameter | ||
estimates and metrics (AIC and BIC) compare with results in DataShop? | estimates and metrics (AIC and BIC) compare with results in DataShop? | ||
2. Modify the regression equation to try to improve the prediction. | 2. Modify the regression equation to try to improve the prediction. | ||
Line 382: | Line 293: | ||
failure opportunities separately (both kinds of opportunities are | failure opportunities separately (both kinds of opportunities are | ||
lumped together in AFM), d) using log of Opportunity, e) including | lumped together in AFM), d) using log of Opportunity, e) including | ||
− | step ( | + | step (as a random effect) ... |
3. Turn in your R file including metrics (log-liklihood, parameters, | 3. Turn in your R file including metrics (log-liklihood, parameters, | ||
AIC, BIC) on the statistical models you compared | AIC, BIC) on the statistical models you compared | ||
4. Summarize whether or not your modification changes model fit (log | 4. Summarize whether or not your modification changes model fit (log | ||
liklihood), changes the number of parameters (from what to what), | liklihood), changes the number of parameters (from what to what), | ||
− | and, most importantly, improves prediction (as measured by AIC or BIC) | + | and, most importantly, improves prediction (e.g., as measured by AIC or BIC or cross validation) |
===== Flex day (Koedinger) ===== | ===== Flex day (Koedinger) ===== | ||
− | *4- | + | *4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics |
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project. | ** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project. | ||
***Option1. More on Educational Data Mining | ***Option1. More on Educational Data Mining | ||
Line 399: | Line 310: | ||
=====Educational Data Mining -- Causal Inference from Data (Scheines) ===== | =====Educational Data Mining -- Causal Inference from Data (Scheines) ===== | ||
− | *4- | + | *4-11 |
− | **Before class on 4- | + | **Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods |
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/ | Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/ | ||
Scroll down and click on the rightmost tab, "Prior work (5)" | Scroll down and click on the rightmost tab, "Prior work (5)" | ||
Line 414: | Line 325: | ||
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research. | **There are also case studies from economics, fMRI, genetics, biology, as well as educational research. | ||
− | *4- | + | *4-13 |
− | |||
− | |||
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]] | **Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]] | ||
− | *4- | + | *4-18 |
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]] | **Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]] | ||
+ | |||
+ | *4-20 NO CLASS - Spring Carnival | ||
=====Experimental Research Methods (Koedinger)===== | =====Experimental Research Methods (Koedinger)===== | ||
− | *4- | + | *4-25 |
− | *4- | + | **First thing: Do "Experimental Methods" Quiz on Blackboard |
− | **Reading: Trochim Ch 7 and 9 | + | **Make progress on your project -- come prepared to tell us about it! |
+ | **Reading: Start Trochim's Ch 7 and 9 | ||
+ | **Optional: Try ANOVA module of OLI Statistics course | ||
+ | **Relevant Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]] | ||
+ | *4-27 | ||
+ | **Reading: Finish Trochim's Ch 7 and 9 | ||
+ | **Optional: Try ANOVA module of OLI Statistics course | ||
**Do two posts on Blackboard. | **Do two posts on Blackboard. | ||
− | * | + | *5-2 |
− | |||
**Reading: Trochim Ch 10 | **Reading: Trochim Ch 10 | ||
− | ** | + | **Relevant Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]] |
− | *5- | + | *5-4 |
**Reading: Trochim Ch 14 | **Reading: Trochim Ch 14 | ||
**Optional: Try ANOVA module of OLI Statistics course | **Optional: Try ANOVA module of OLI Statistics course | ||
Line 438: | Line 354: | ||
If needed, schedule a course wrap-up | If needed, schedule a course wrap-up | ||
− | Final project is due May | + | Final project is due May 11. |
Latest revision as of 15:47, 23 April 2017
Contents
- 1 Research Methods for the Learning Sciences 05-748
- 2 Goals
- 3 Course Prerequisites
- 4 Textbook and Readings
- 5 Flipped Homework: Reading Reports and Pre-Class Assignments
- 6 Grading
- 7 Class Schedule in Brief
- 8 Class Schedule with Readings and Assignments
- 8.1 Course Intro, Research Questions, Picking Methods (Koedinger)
- 8.2 Video and Verbal Protocol Analysis (Lovett, Rosé)
- 8.3 Cognitive Task Analysis (CTA) (Koedinger)
- 8.4 Psychometrics, Reliability, Item Response Theory (Nugent)
- 8.5 NO CLASS – Spring break 3-14 and 3-16
- 8.6 Surveys, Questionnaires, Interviews (Ogan)
- 8.7 Educational Data Mining -- Learning Curve Analysis (Koedinger)
- 8.8 Flex day (Koedinger)
- 8.9 Educational Data Mining -- Causal Inference from Data (Scheines)
- 8.10 Experimental Research Methods (Koedinger)
- 8.11 Wrap-up
Research Methods for the Learning Sciences 05-748
Spring 2017 Syllabus Carnegie Mellon University
Class times
4:30 to 5:50 Tuesday & Thursday
Location
4101 Gates/Hillman
Instructor
Professor Ken Koedinger Office: 3601 Newell-Simon Hall, Phone: 412-268-7667 Email: Koedinger@cmu.edu, Office hours by appointment
Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines
Class URLs
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014 learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
For reading reports: www.cmu.edu/blackboard
Goals
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.
Course Prerequisites
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.
Textbook and Readings
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.
Find it by googling for the title or clicking here.
Other readings will be assigned in class. See below.
Flipped Homework: Reading Reports and Pre-Class Assignments
We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.
Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard (www.cmu.edu/blackboard) for this purpose.
Unless otherwise directed by instructors, students should make two posts on the readings before 3:30pm on the day of class that those readings are due. If slides for the class are available, please review these as well.
These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!
In general, please come to class prepared to ask questions and give answers.
Your two posts may be original or in response to another post (one of both is nice).
- Original posts should contain one or more of the following:
- something you learned from the reading or slides
- a question you have about the reading or slides or about the topic in general
- a connection with something you learned or did previously in this or another course, or in other professional work or research
- Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.
You may be asked to do other activities before class, such as answer questions on-line using the Assistment system, parts of the an OLI course, or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.
Grading
There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.
- Course work
- 30% Before-class preparation, including reading reports, and in-class participation
- 40% Assignments
- Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
- 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
- Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
- No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.
Class Schedule in Brief
- Formulating Good Research Questions: Jan 17 (T)
- Choosing Qualitative & Quantitative Methods: Jan 19 (R)
- Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
- Guest Instructors: Marsha Lovett & Carolyn Rose
- Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
- Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
- Guest Instructor: Rebecca Nugent
- Cognitive Task Analysis - Quantitative: Mar 9 (T)
- NO CLASS – Spring break, Mar 14, 16 (TR)
- Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
- Guest Instructor: Amy Ogan
- Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
- Flex day (Educational Design Research?): Apr 6 (R)
- Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
- Guest Instructor: Richard Scheines
- NO CLASS – Spring Carnival, Apr 20 (R)
- Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
- Wrap-up: May 9 (T)
Class Schedule with Readings and Assignments
NOTE: This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.
Course Intro, Research Questions, Picking Methods (Koedinger)
- 1-17
- Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- but here's Chapter 1.
- Do the chpt 1 quiz
- Lecture slides 2014
- 1-19 Choosing Qualitative & Quantitative Methods
- Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time here's Chapter 6 if you need it.
- Do the chpt 6 quiz
- Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). Instructional complexity and the science to constrain it. Science, 342, 935-937. PDF
- [Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. PDF
- Draft Table relating research purposes and methods: [1]
Video and Verbal Protocol Analysis (Lovett, Rosé)
The plan for this session and readings are in this zip file, which is also available on blackboard.
Cognitive Task Analysis (CTA) (Koedinger)
- 2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
- Clark et al (2012) on Cognitive Task Analysis and improving instruction
- Do two posts on Blackboard.
- One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
- Optional readings:
- 2-16 Rational CTA via Cognitive Modeling
- Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). PNAS-1996-Zhu-Simon.pdf
- [Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. HowPeopleLearnCh2.pdf
- Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
- [Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. Zhu&Simon-1987.pdf
- 2-21 Doing CTA for higher-level thinking/learning skills
- Azevedo et al on think alouds during learning from hypermedia AzevedoMoosJohnson&Chauncey2010.pdf
- Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. AlevenITS2004.pdf
- Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. Klahr&carver88.pdf
- Pick one of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
- Other possible readings:
- Kinds of CTA and instructional design: Lovett Lovett01CandI.pdf
- Relevant to cognitive modeling: Newell & Simon Human_Problem_Solving.pdf
- A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. Siegler76.pdf
- 3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
- Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences, 13 (2), 129-164. Koedinger-Nathan-LS04.pdf
- In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
- Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html KoedingerMacLaren02.pdf
- Do two posts on these readings.
- Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences, 13 (2), 129-164. Koedinger-Nathan-LS04.pdf
- Other optional readings
- See prior CTA assignment.
- Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. (pp. 471-476.) Austin, TX: Cognitive Science Society. Koedinger-mclaughlin-cs2010.pdf
- Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. Rittle-Johnson-Koedinger-cogsci01.pdf
- Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science. KLI-paper-v5.13.pdf
- Other optional readings
Psychometrics, Reliability, Item Response Theory (Nugent)
- TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.
- 2-23
- Quick introduction to the R statistical language
- Please complete and bring comments & questions to class on Tues Feb 28.
- Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
- R-preassignment.pdf - instructions for this assignment
- r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
- thermo11_data_integrated.csv - a data set for the examples.
- 2-28
1. From Trochim:
A. Chapter 3 - the vocabulary of measurement B. Chapter 5 - on constructing scales (it's ok to focus on the material up through sect 5.2a; the rest is more of a skim [but I'd be happy to talk about that in class also])
2. On item response theory (IRT), a set of statistical models that are used to construct scales and to derive scores from them, especially in education and psychological research:
A. Harris Article (PDF) Please take and self-score the test at the end of this article. Count each part of question one as one point, and each of the remaining three questions as one point (no partial credit!). Bring your 8 scores to class. E.g. if you missed 1(c) and (d), and you also missed question 4, then you would bring to class the following scores: 1 1 0 0 1 1 1 0 If you missed 1(a) and (b) and question 2, bring the following scores: 0 0 1 1 1 0 1 1 (note that the total score is 5 in both cases, but the pattern of rights and wrongs differs; it is the pattern that we are interested in). B. Please browse *online* through pp 1-23 of the pdf at [2]. The math is a bit heavy going but there are links to apps that illustrate various points in the harris article. So skim the math and play with the apps.
- 3-2
The assignment for this lecture has two parts.
- (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
- (B) The readings below.
On Tue we will discuss whatever of A and/or B seem interesting
1. "Psychometric Principles in Student Assessment" by Mislevy et al (Mislevy (PDF))
Read through p 18. This is a more modern modern look at some of the same issues that are addressed in Trochim's chapters. The remainder of this paper surveys various probabilistic models for the "measurement model" portion of Mislevy's framework (Figure 1). It is quite interesting but we will not pursue it.
2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma (Junker, Sijtsma (PDF))
Please read up through p 266 only. The math is a bit heavy going so please try to read around it to see what the point of the article is. We will try to look at some of the data in the article as examples in lecture 2.
- 3-7 Continued discussion of Psychometrics
NO CLASS – Spring break 3-14 and 3-16
Surveys, Questionnaires, Interviews (Ogan)
- 3-21
- Reading: Trochim Ch 4 and 5
- You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
- Reading: Trochim Ch 4 and 5
- 3-23
- Do the following homework assignment Media:Arm-modQuestEduc.doc. Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
- Readings
- Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. Media:Tourangeau_SensitiveQuestions.pdf
- Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. Media:Tourangeau_RememberingWhatHappened.pdf
Educational Data Mining -- Learning Curve Analysis (Koedinger)
- 3-28
- BRING YOUR LAPTOP FOR ALL THESE SESSIONS
- Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday).
- Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
- Read the following paper and make two posts as usual.
- Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer. Stamper-Koedinger-AIED2011.pdf
- Optional:Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. RittterSchooler01.pdf
- Assignment: The assignment ( Learning-curve-assignment-2014.doc) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) comfortably before class on Thursday -- by 3pm. Also, in addition to the problem content file indicated in the assignment handout see other files in the same location to get a more complete description and list of the files: Geometry Area Problems PDF Explanation.docx and solutions.zip.
- 3-30
- Read the following paper and make two posts as usual.
- Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. KoedingerMcLaughlinStamperEDM12.pdf
- In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (at least get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-5.
- Read the following paper and make two posts as usual.
- 4-4
- In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
- Read the following paper and make two posts as usual.
- Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA AIED2007_EDM_Zhang_ld_transfer.pdf
- Optional: Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. Media:2000_roberts_pashler.pdf
- Optional: Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. Media:GOF.doc
Do A or B: A. Modify a KC model in a DataShop dataset 1. What is the DataShop dataset you modified? (Look for datasets with the lego block icon on them -- these have associated problem descriptions) 2. Describe how you used the HMST procedure (from Stamper paper) to identify a KC to try to improve 3. Show how you recoded that KC with new KCs (turn in your modified KC file) & describe why you made the change you did 4. After importing your new KC model to DataShop, did it improve the predictions on any of the metrics, AIC, BIC, or cross validation? (Caution: Make sure your new KC model labels the same number of observations as the KC model you are modifying.)
B. Use R to create an alternative statistical model to AFM 1. Approximate AFM in R using either glm or glmer (in package lme4). You can find R code that mimics AFM in the DataShop help, here: https://pslcdatashop.web.cmu.edu/help?page=rSoftware
How do the parameter estimates and metrics (AIC and BIC) compare with results in DataShop? 2. Modify the regression equation to try to improve the prediction. Some options include: a) adding a student by KC interaction (there are just main effects of student and KC in AFM), b) adding student slopes (there is just a KC slope in AFM), c) counting success and failure opportunities separately (both kinds of opportunities are lumped together in AFM), d) using log of Opportunity, e) including step (as a random effect) ... 3. Turn in your R file including metrics (log-liklihood, parameters, AIC, BIC) on the statistical models you compared 4. Summarize whether or not your modification changes model fit (log liklihood), changes the number of parameters (from what to what), and, most importantly, improves prediction (e.g., as measured by AIC or BIC or cross validation)
Flex day (Koedinger)
- 4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
- We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
- Option1. More on Educational Data Mining
- Option2. Return to Design Research & Qualitative Methods (Koedinger)
- Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
- Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). PDF
- Optional reading: Chapter on Design Research in Handbook of Learning Sciences
- We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
Educational Data Mining -- Causal Inference from Data (Scheines)
- 4-11
- Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/ Scroll down and click on the rightmost tab, "Prior work (5)" Click on "Empirical Research Methods" and then on "[Enter Course]" Click on "CMU users sign in here" to login with your CMU account or "Enter Without an Account" Complete "UNIT 2: Regression, Prediction and Causation"
- See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
- Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
- a) Tutorial on causal learning (my tutorial on Tetrad)
- b) Educational Research I (overview of causal discovery in educational research)
- c) Educational Research II (Martina explaining the paper you are assigned)
- There are also case studies from economics, fMRI, genetics, biology, as well as educational research.
- 4-13
- Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. PDF
- 4-18
- Read and post about Rau, Scheines, Aleven, & Rummel (2013
- 4-20 NO CLASS - Spring Carnival
Experimental Research Methods (Koedinger)
- 4-25
- First thing: Do "Experimental Methods" Quiz on Blackboard
- Make progress on your project -- come prepared to tell us about it!
- Reading: Start Trochim's Ch 7 and 9
- Optional: Try ANOVA module of OLI Statistics course
- Relevant Slides: Experimental_Methods.ppt and True-Experiments.ppt
- 4-27
- Reading: Finish Trochim's Ch 7 and 9
- Optional: Try ANOVA module of OLI Statistics course
- Do two posts on Blackboard.
- 5-2
- Reading: Trochim Ch 10
- Relevant Slides: Quasi-Experiments.ppt
- 5-4
- Reading: Trochim Ch 14
- Optional: Try ANOVA module of OLI Statistics course
Wrap-up
If needed, schedule a course wrap-up
Final project is due May 11.