Educational Research Methods 2017 - Revision history

Koedinger: /* Experimental Research Methods (Koedinger) */

2017-04-23T15:47:35Z

Experimental Research Methods (Koedinger)

← Older revision		Revision as of 15:47, 23 April 2017
Line 334:		Line 334:

	=====Experimental Research Methods (Koedinger)=====		=====Experimental Research Methods (Koedinger)=====
	*4-25 ~~Continuation~~ of ~~Causal Inference~~		*4-25
			**First thing: Do "Experimental Methods" Quiz on Blackboard
			**Make progress on your project -- come prepared to tell us about it!
			**Reading: Start Trochim's Ch 7 and 9
			**Optional: Try ANOVA module of OLI Statistics course
			**Relevant Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt\|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt\|True-Experiments.ppt]]
	*4-27		*4-27
	**Reading: Trochim Ch 7 and 9		**Reading: Finish Trochim's Ch 7 and 9
			**Optional: Try ANOVA module of OLI Statistics course
	**Do two posts on Blackboard.		**Do two posts on Blackboard.
	**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt\|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt\|True-Experiments.ppt]]
	*5-2		*5-2
	**Reading: Trochim Ch 10		**Reading: Trochim Ch 10
	**~~OLD~~ Slides: [[Media:L04-quasi-experiments.ppt\|Quasi-Experiments.ppt]]		**Relevant Slides: [[Media:L04-quasi-experiments.ppt\|Quasi-Experiments.ppt]]
	*5-4		*5-4
	**Reading: Trochim Ch 14		**Reading: Trochim Ch 14

Koedinger: /* Educational Data Mining -- Learning Curve Analysis (Koedinger) */

2017-03-30T12:09:58Z

Educational Data Mining -- Learning Curve Analysis (Koedinger)

Koedinger: /* Educational Data Mining -- Learning Curve Analysis (Koedinger) */

2017-03-29T20:54:45Z

Educational Data Mining -- Learning Curve Analysis (Koedinger)

Koedinger: /* Educational Data Mining -- Learning Curve Analysis (Koedinger) */

2017-03-29T15:11:53Z

Educational Data Mining -- Learning Curve Analysis (Koedinger)

← Older revision		Revision as of 15:11, 29 March 2017
Line 256:		Line 256:
	***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf\| Stamper-Koedinger-AIED2011.pdf]]		***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf\| Stamper-Koedinger-AIED2011.pdf]]
	***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf \| RittterSchooler01.pdf]]		***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf \| RittterSchooler01.pdf]]
	**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc \| Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) ~~by 9am~~ on Thursday.		**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc \| Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) comfortably before class on Thursday -- by 3pm. Also, in addition to the problem content file indicated in the assignment handout see other files in the same location to get a more complete description and list of the files: Geometry Area Problems PDF Explanation.docx and solutions.zip.
	*3-30		*3-30
	**Read the following paper and make two posts as usual.		**Read the following paper and make two posts as usual.

Koedinger: /* Surveys, Questionnaires, Interviews (Ogan) */

2017-03-22T20:25:46Z

Surveys, Questionnaires, Interviews (Ogan)

← Older revision		Revision as of 20:25, 22 March 2017
Line 239:		Line 239:

	=====Surveys, Questionnaires, Interviews (Ogan) =====		=====Surveys, Questionnaires, Interviews (Ogan) =====
	* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
	*3-21		*3-21
	**Reading: Trochim Ch 4 and 5		**Reading: Trochim Ch 4 and 5
	***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.		***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
	*3-23		*3-23
	**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. ~~Sara directs:~~ Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.		**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
	**Readings		**Readings
	***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]		***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]

Koedinger: /* Psychometrics, reliability, Item Response Theory (Nugent) */

2017-02-14T23:43:41Z

Psychometrics, reliability, Item Response Theory (Nugent)

← Older revision		Revision as of 23:43, 14 February 2017
Line 147:		Line 147:
	***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf\|KLI-paper-v5.13.pdf]]		***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf\|KLI-paper-v5.13.pdf]]

	=====Psychometrics, ~~reliability~~, Item Response Theory (Nugent)=====		=====Psychometrics, Reliability, Item Response Theory (Nugent)=====

	* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.		* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

	*2-28		*2-23

	**Quick introduction to the R statistical language		**Quick introduction to the R statistical language
Line 161:		Line 161:
	*** thermo11_data_integrated.csv - a data set for the examples.		*** thermo11_data_integrated.csv - a data set for the examples.

	*3-2		*2-28

	1. From Trochim:		1. From Trochim:
Line 206:		Line 206:
	So skim the math and play with the apps.		So skim the math and play with the apps.

	*3-7		*3-2

	The assignment for this lecture has two parts.		The assignment for this lecture has two parts.
Line 234:		Line 234:
	in lecture 2.		in lecture 2.

	*3-9 Continued discussion of Psychometrics ~~[moved Design Research as option for Flex Day]~~		*3-7 Continued discussion of Psychometrics

	=====NO CLASS – Spring break 3-14 and 3-16 =====		=====NO CLASS – Spring break 3-14 and 3-16 =====

Koedinger: /* Cognitive Task Analysis (CTA) (Koedinger) */

2017-02-14T23:42:15Z

Cognitive Task Analysis (CTA) (Koedinger)

← Older revision		Revision as of 23:42, 14 February 2017
Line 135:		Line 135:
	**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf\|Siegler76.pdf]]		**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf\|Siegler76.pdf]]

	*2-23 Empirical quantitative CTA via Difficulty Factors Assessment		*3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
	**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf\|Koedinger-Nathan-LS04.pdf]]		**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf\|Koedinger-Nathan-LS04.pdf]]
	***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.		***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.

Koedinger: /* Class Schedule in Brief */

2017-02-14T23:37:12Z

Class Schedule in Brief

← Older revision		Revision as of 23:37, 14 February 2017
Line 74:		Line 74:
	* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)		* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
	** Guest Instructors: Marsha Lovett & Carolyn Rose		** Guest Instructors: Marsha Lovett & Carolyn Rose
	* Performing Cognitive Task Analysis: Feb 14, 16, 21~~, 23 (TRTR)~~		* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
	* Educational Design Research: Feb 28 (T)		* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
	* Educational Measurement & Psychometrics: Mar 2, 7~~, 9,~~ (~~RTR~~)		** Guest Instructor: Rebecca Nugent
	** Guest Instructor: ~~Brian Junker~~		* Cognitive Task Analysis - Quantitative: Mar 9 (T)
	* NO CLASS – Spring break, Mar 14, 16 (TR)		* NO CLASS – Spring break, Mar 14, 16 (TR)
	* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)		* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
	** Guest Instructor: ~~Sara Kiesler~~		** Guest Instructor: Amy Ogan
	* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)		* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
	* Flex day: Apr 6 (R)		* Flex day (Educational Design Research?): Apr 6 (R)
	* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)		* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
	** Guest Instructor: Richard Scheines		** Guest Instructor: Richard Scheines

Koedinger: /* Psychometrics, reliability, Item Response Theory (Nugent) */

2017-02-06T16:44:57Z

Psychometrics, reliability, Item Response Theory (Nugent)

← Older revision		Revision as of 16:44, 6 February 2017
Line 147:		Line 147:
	***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf\|KLI-paper-v5.13.pdf]]		***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf\|KLI-paper-v5.13.pdf]]

	=====Psychometrics, reliability, Item Response Theory (~~Junker~~)=====		=====Psychometrics, reliability, Item Response Theory (Nugent)=====

	*~~NEW ASSIGNMENTS [~~Plans for these classes ~~were~~ communicated by ~~Brian Junker via email~~.]		* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

	*2-28		*2-28

Koedinger: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

2017-01-24T22:24:06Z

Video and Verbal Protocol Analysis (Lovett, Rosé)

@@ Line 108: / Line 108: @@
 =====Video and Verbal Protocol Analysis (Lovett, Rosé) =====
-[TO BE UPDATED]
+The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.
 =====Cognitive Task Analysis (CTA) (Koedinger) =====

← Older revision		Revision as of 12:09, 30 March 2017
Line 269:		Line 269:

	Do A or B:		Do A or B:
	A. Modify a KC model in a DataShop dataset		A. Modify a KC model in a DataShop dataset
	1. What is the DataShop dataset you modified?		1. What is the DataShop dataset you modified? (Look for datasets with the lego block icon on them -- these have associated problem descriptions)
	2. Describe how you used the HMST procedure (from Stamper paper)		2. Describe how you used the HMST procedure (from Stamper paper)
	to identify a KC to try to improve		to identify a KC to try to improve
Line 276:		Line 276:
	KC file) & describe why you made the change you did		KC file) & describe why you made the change you did
	4. After importing your new KC model to DataShop, did it improve the		4. After importing your new KC model to DataShop, did it improve the
	predictions ~~(are~~ any of the metrics, AIC, BIC, or cross validation)?		predictions on any of the metrics, AIC, BIC, or cross validation?
	(Caution: Make sure your new KC model labels the same number of		(Caution: Make sure your new KC model labels the same number of
	observations as the KC model you are modifying.)		observations as the KC model you are modifying.)

	B. Use R to create an alternative statistical model to AFM		B. Use R to create an alternative statistical model to AFM
	1. Approximate AFM in R using either glm or ~~lmer~~. How do the parameter		1. Approximate AFM in R using either glm or glmer (in package lme4). You
			can find R code that mimics AFM in the DataShop help, here:
			https://pslcdatashop.web.cmu.edu/help?page=rSoftware

			How do the parameter
	estimates and metrics (AIC and BIC) compare with results in DataShop?		estimates and metrics (AIC and BIC) compare with results in DataShop?
	2. Modify the regression equation to try to improve the prediction.		2. Modify the regression equation to try to improve the prediction.
Line 289:		Line 293:
	failure opportunities separately (both kinds of opportunities are		failure opportunities separately (both kinds of opportunities are
	lumped together in AFM), d) using log of Opportunity, e) including		lumped together in AFM), d) using log of Opportunity, e) including
	step (~~perhaps~~ as a random effect) ...		step (as a random effect) ...
	3. Turn in your R file including metrics (log-liklihood, parameters,		3. Turn in your R file including metrics (log-liklihood, parameters,
	AIC, BIC) on the statistical models you compared		AIC, BIC) on the statistical models you compared
	4. Summarize whether or not your modification changes model fit (log		4. Summarize whether or not your modification changes model fit (log
	liklihood), changes the number of parameters (from what to what),		liklihood), changes the number of parameters (from what to what),
	and, most importantly, improves prediction (as measured by AIC or BIC)		and, most importantly, improves prediction (e.g., as measured by AIC or BIC or cross validation)

	===== Flex day (Koedinger) =====		===== Flex day (Koedinger) =====

← Older revision		Revision as of 20:54, 29 March 2017
Line 251:		Line 251:
	*3-28		*3-28
	**BRING YOUR LAPTOP FOR ALL THESE SESSIONS		**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
	**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday ~~by 9am~~).		**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday).
	***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.		***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
	**Read the following paper and make two posts as usual.		**Read the following paper and make two posts as usual.
Line 260:		Line 260:
	**Read the following paper and make two posts as usual.		**Read the following paper and make two posts as usual.
	***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf\|KoedingerMcLaughlinStamperEDM12.pdf]]		***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf\|KoedingerMcLaughlinStamperEDM12.pdf]]
	**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.		**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (at least get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-5.
	*4-4		*4-4
	**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).		**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
Line 281:		Line 281:

	B. Use R to create an alternative statistical model to AFM		B. Use R to create an alternative statistical model to AFM
	1. Approximate ~~afm~~ in R using either glm or lmer. How do the parameter		1. Approximate AFM in R using either glm or lmer. How do the parameter
	estimates and metrics (AIC and BIC) compare with results in DataShop?		estimates and metrics (AIC and BIC) compare with results in DataShop?
	2. Modify the regression equation to try to improve the prediction.		2. Modify the regression equation to try to improve the prediction.