Theory Wiki - User contributions [en]

Main Page

2019-09-05T12:48:20Z

Koedinger: /* Robust Learning Theoretical Framework */

== Robust Learning Theoretical Framework ==

A key goal of LearnLab is to support learning scientists in providing explanations of results using, as much as possible, the same core terminology and addressing an accumulating body of precise theoretical [[Instructional Principles and Hypotheses|principles of instruction]]. While a single theory of learning may emerge in the long term, the immediate goal is to encourage researchers to maximize the overlap between each others' theories. We want to help the field get beyond the "[[Toothbrush Problem]]" in theorizing.

In 2012, we published a theoretical framework, called the [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Knowledge-Learning-Instruction (KLI) framework]. This framework builds on our 2006 [http://learnlab.org/clusters/PSLC_Theory_Frame_June_15_2006.pdf theoretical framework document] and, importantly, on the contributions to this wiki. In 2013, we published [http://pact.cs.cmu.edu/pubs/Koedinger-Science-2013.pdf a paper in Science] reviewing instructional principles, the complexity of their combinations, and recommendations for LearnLab style research to address this complexity.

The contents of this wiki can be accessed through multiple pathways including 1) an outline of [[Instructional Principles and Hypotheses|Instructional Principles]], 2) a set of early LearnLab research cluster page: [[Coordinative Learning]], [[Interactive Communication]], and [[Refinement and Fluency]], and 3) a later set of LearnLab research thrust pages: [[Cognitive Factors]] [[Metacognition and Motivation]], [[Social Communication]], and [[Computational Modeling and Data Mining]]. See also these summaries: [https://learnlab.org/index.php/cognitive-factors-research-thrust/ Cognitive Factors], [https://learnlab.org/index.php/metacognition-and-motivation-research-thrust/ Metacognition and Motivation], [https://learnlab.org/index.php/social-communication-research-thrust/ Social Communication], and [https://learnlab.org/index.php/computational-modeling-and-data-mining-research-thrust/ Computational Modeling and Data Mining].

Many other on-line lists of instructional and learning principles exist ['''''please add more!''''']:
* The principles below are summarized in this [https://science.sciencemag.org/content/342/6161/935 Instructional Complexity] paper by LearnLab Director Ken Koedinger and colleagues. Thirty instructional principles are merged from lists below as shown here: [[File:Instructional_Principles_Table.xlsx]]. [http://edugames.design/principles An interactive website] provides a quick overview of these 30 instructional principles.
* [http://ies.ed.gov/ncer/pubs/practiceguides/20072004.asp Organizing Instruction and Study to Improve Student Learning], one of the Practice Guides of the Department of Education, Institute for Education Sciences. See also:
** [http://www.nctq.org/dmsStage/Learning_About_Learning_Report A web-based update of this guide]
* [http://www.psyc.memphis.edu/learning/principles/ Principles of Learning] from Lifelong Learning at Work and at Home, a subgroup of the Association for Psychological Science.
* [http://www.edu-design-principles.org Design Principles Database] maintained by the NSF-funded [http://www.telscenter.org/ TELS (Technology Enhanced Learning in Science)] project
* [http://www.cmu.edu/teaching/principles/ Principles of Teaching and Learning] from CMU's Eberly Center for Teaching and Learning
* [http://en.wikipedia.org/wiki/Principles_of_learning Wikipedia entry for principles of learning]
* [http://www.udlcenter.org/aboutudl/udlguidelines Universal Design for Learning guidelines]
* See a review of principles and the role of technology in refining them in [https://www.youtube.com/watch?v=ysWTEWx9L_A this talk by Art Graesser].

[http://pact.cs.cmu.edu/koedinger/Koedinger10_WMV%20V9.wmv Ken Koedinger HCII Seminar Talk: Why Designing Effective Learning Interactions Is Not Easy and How We Can Do Better: Part I]

=== Toward a Hierarchical Structure of Robust Learning Hypotheses and Findings ===

The structure of the research cluster and study pages is as follows (a template can be found at [[Project Page Template and Creation Instructions]]). When a set of explanations share many terms and hypotheses, we make a node for each explanation, make a node for their common features, and link the nodes so that the common-feature node is the parent of each explanation node. In most cases a "node" is a single wiki page, but sometimes a node involves several wiki pages.

In order to more clearly display the integration, each node contains:
#An ''abstract'' that briefly describes the research encompassed by the node;
#A ''glossary'' that defines terms used elsewhere in this node but not defined in the nodes that are parents, grandparents, etc. of this node;
#The ''research question'' stated as concisely as possible, usually in a single sentence;
#A ''background and significance'' section that briefly summarizes prior work on the research question and why it is important to answer it;
#The ''dependent variables'', which are observable and typically measure competence, motivation, interaction, meta-learning, or some other pedagogically desirable outcome;
#The ''independent variables'', which typically include instructional environment, activity or method (the instructional "treatment" vs. "control"), and perhaps some student individual difference variables, such as gender or first language;
#The ''hypothesis'', which is a concise statement of the relationship among the variables that answers the research question;
#The ''findings'', which are the results of the study if it has been performed or the expected findings from the study if it has not -- explicitly indicate if the findings are preliminary;
#An ''explanation'' that describes the theoretical rationale for the hypothesis using the PSLC theoretical framework. It should be a paragraph or two and provide a causal chain that mentions mediating variables -- unobservable, hypothetical attributes of the students (e.g., knowledge components or path choices), how the treatment affects these, and how they, in turn, affect the dependent variables;
#The ''descendants'', which lists links to descendant nodes of this one, if there are any;
#A ''further information'' section that points to documents using hyper links and/or references in APA format. Each indicates briefly the document's relationship to the node (e.g., whether the document is a paper reporting the node in full detail, a proposal describing the motivation and design of the study in more detail, the node for a similar PSLC research study, etc.).

Experience suggests that the glossaries carry much of the load in explaining the research, and that carefully defining and exemplifying terms often pays off later in reducing confusion and facilitating collaboration. Consequently, the glossaries are sometimes so long that they are spit off as separate wiki pages.

The [[Root_node|root node of the hierarchy]] represents the overarching research question of how to achieve robust learning particularly in academic settings, like K12 schools and colleges. The question is necessarily abstract and is not the sort of question that can actually be tested by a single decisive experiment.

The immediate descendants of the root node are three nodes representing somewhat more specific research questions. There are nodes for each of [[Coordinative Learning]], [[Interactive Communication]] and [[Refinement and Fluency]]. These present somewhat more concrete research questions. They are specializations to the overarching questions, and form a bridge to testable hypotheses posed by individual research projects.

The leaves of the hierarchy (i.e., nodes with no descendants) represent individual research studies. A leaf node can also represent a group of studies or a whole project if the activities are sufficiently similar that it makes sense to summarize them with a single node. Each leaf node is maintained by its project’s leader and may or may not be publicly accessible depending on the state of the research.

Between the cluster nodes and the leaves, there may be some intervening nodes. For instance, if a group of Coordinative Learning studies all address a similar research question (e.g., how to use verbal and visual instruction together effectively), then a node may be created to summarize their shared aspects. Its parent is the Coordinative Learning cluster node, and its descendants are the relevant project nodes.

For a draft of [[Macro-level framework|an alternative framework based on independent dimensions of instruction click here]].

Main Page

2019-09-05T12:29:14Z

Koedinger: /* Robust Learning Theoretical Framework */

== Robust Learning Theoretical Framework ==

A key goal of LearnLab is to support learning scientists in providing explanations of results using, as much as possible, the same core terminology and addressing an accumulating body of precise theoretical [[Instructional Principles and Hypotheses|principles of instruction]]. While a single theory of learning may emerge in the long term, the immediate goal is to encourage researchers to maximize the overlap between each others' theories. We want to help the field get beyond the "[[Toothbrush Problem]]" in theorizing.

In 2012, we published a theoretical framework, called the [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Knowledge-Learning-Instruction (KLI) framework]. This framework builds on our 2006 [http://learnlab.org/clusters/PSLC_Theory_Frame_June_15_2006.pdf theoretical framework document] and, importantly, on the contributions to this wiki. In 2013, we published [http://pact.cs.cmu.edu/pubs/Koedinger-Science-2013.pdf a paper in Science] reviewing instructional principles, the complexity of their combinations, and recommendations for LearnLab style research to address this complexity.

In order to display the integration across research projects, this wiki maintains multiple theoretical hierarchies, one of [[Instructional Principles and Hypotheses|Instructional Principles]] and another of empirical studies, which are found on the cluster pages: [[Coordinative Learning]], [[Interactive Communication]], and [[Refinement and Fluency]].

Many other on-line lists of instructional and learning principles exist ['''''please add more!''''']:
* [http://ies.ed.gov/ncer/pubs/practiceguides/20072004.asp Organizing Instruction and Study to Improve Student Learning], one of the Practice Guides of the Department of Education, Institute for Education Sciences. See also:
** [http://www.nctq.org/dmsStage/Learning_About_Learning_Report A web-based update of this guide]
* [http://www.psyc.memphis.edu/learning/principles/ Principles of Learning] from Lifelong Learning at Work and at Home, a subgroup of the Association for Psychological Science.
* [http://www.edu-design-principles.org Design Principles Database] maintained by the NSF-funded [http://www.telscenter.org/ TELS (Technology Enhanced Learning in Science)] project
* [http://www.cmu.edu/teaching/principles/ Principles of Teaching and Learning] from CMU's Eberly Center for Teaching and Learning
* [http://en.wikipedia.org/wiki/Principles_of_learning Wikipedia entry for principles of learning]
* [http://www.udlcenter.org/aboutudl/udlguidelines Universal Design for Learning guidelines]
* See a review of principles and the role of technology in refining them in [https://www.youtube.com/watch?v=ysWTEWx9L_A this talk by Art Graesser].
* [http://edugames.design/principles An interactive website] describes the principles from our [https://science.sciencemag.org/content/342/6161/935 instructional complexity] paper which includes 30 principles merged from lists above as shown in this table: [[File:Instructional_Principles_Table.xlsx]].

[http://pact.cs.cmu.edu/koedinger/Koedinger10_WMV%20V9.wmv Ken Koedinger HCII Seminar Talk: Why Designing Effective Learning Interactions Is Not Easy and How We Can Do Better: Part I]

=== Toward a Hierarchical Structure of Robust Learning Hypotheses and Findings ===

The structure of the research cluster and study pages is as follows (a template can be found at [[Project Page Template and Creation Instructions]]). When a set of explanations share many terms and hypotheses, we make a node for each explanation, make a node for their common features, and link the nodes so that the common-feature node is the parent of each explanation node. In most cases a "node" is a single wiki page, but sometimes a node involves several wiki pages.

In order to more clearly display the integration, each node contains:
#An ''abstract'' that briefly describes the research encompassed by the node;
#A ''glossary'' that defines terms used elsewhere in this node but not defined in the nodes that are parents, grandparents, etc. of this node;
#The ''research question'' stated as concisely as possible, usually in a single sentence;
#A ''background and significance'' section that briefly summarizes prior work on the research question and why it is important to answer it;
#The ''dependent variables'', which are observable and typically measure competence, motivation, interaction, meta-learning, or some other pedagogically desirable outcome;
#The ''independent variables'', which typically include instructional environment, activity or method (the instructional "treatment" vs. "control"), and perhaps some student individual difference variables, such as gender or first language;
#The ''hypothesis'', which is a concise statement of the relationship among the variables that answers the research question;
#The ''findings'', which are the results of the study if it has been performed or the expected findings from the study if it has not -- explicitly indicate if the findings are preliminary;
#An ''explanation'' that describes the theoretical rationale for the hypothesis using the PSLC theoretical framework. It should be a paragraph or two and provide a causal chain that mentions mediating variables -- unobservable, hypothetical attributes of the students (e.g., knowledge components or path choices), how the treatment affects these, and how they, in turn, affect the dependent variables;
#The ''descendants'', which lists links to descendant nodes of this one, if there are any;
#A ''further information'' section that points to documents using hyper links and/or references in APA format. Each indicates briefly the document's relationship to the node (e.g., whether the document is a paper reporting the node in full detail, a proposal describing the motivation and design of the study in more detail, the node for a similar PSLC research study, etc.).

Experience suggests that the glossaries carry much of the load in explaining the research, and that carefully defining and exemplifying terms often pays off later in reducing confusion and facilitating collaboration. Consequently, the glossaries are sometimes so long that they are spit off as separate wiki pages.

The [[Root_node|root node of the hierarchy]] represents the overarching research question of how to achieve robust learning particularly in academic settings, like K12 schools and colleges. The question is necessarily abstract and is not the sort of question that can actually be tested by a single decisive experiment.

The immediate descendants of the root node are three nodes representing somewhat more specific research questions. There are nodes for each of [[Coordinative Learning]], [[Interactive Communication]] and [[Refinement and Fluency]]. These present somewhat more concrete research questions. They are specializations to the overarching questions, and form a bridge to testable hypotheses posed by individual research projects.

The leaves of the hierarchy (i.e., nodes with no descendants) represent individual research studies. A leaf node can also represent a group of studies or a whole project if the activities are sufficiently similar that it makes sense to summarize them with a single node. Each leaf node is maintained by its project’s leader and may or may not be publicly accessible depending on the state of the research.

Between the cluster nodes and the leaves, there may be some intervening nodes. For instance, if a group of Coordinative Learning studies all address a similar research question (e.g., how to use verbal and visual instruction together effectively), then a node may be created to summarize their shared aspects. Its parent is the Coordinative Learning cluster node, and its descendants are the relevant project nodes.

For a draft of [[Macro-level framework|an alternative framework based on independent dimensions of instruction click here]].

Main Page

2019-09-04T14:41:30Z

Koedinger: /* Robust Learning Theoretical Framework */

== Robust Learning Theoretical Framework ==

A key goal of LearnLab is to support learning scientists in providing explanations of results using, as much as possible, the same core terminology and addressing an accumulating body of precise theoretical [[Instructional Principles and Hypotheses|principles of instruction]]. While a single theory of learning may emerge in the long term, the immediate goal is to encourage researchers to maximize the overlap between each others' theories. We want to help the field get beyond the "[[Toothbrush Problem]]" in theorizing.

In 2012, we published a theoretical framework, called the [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Knowledge-Learning-Instruction (KLI) framework]. This framework builds on our 2006 [http://learnlab.org/clusters/PSLC_Theory_Frame_June_15_2006.pdf theoretical framework document] and, importantly, on the contributions to this wiki. In 2013, we published [http://pact.cs.cmu.edu/pubs/Koedinger-Science-2013.pdf a paper in Science] reviewing instructional principles, the complexity of their combinations, and recommendations for LearnLab style research to address this complexity.

In order to display the integration across research projects, this wiki maintains multiple theoretical hierarchies, one of [[Instructional Principles and Hypotheses|Instructional Principles]] and another of empirical studies, which are found on the cluster pages: [[Coordinative Learning]], [[Interactive Communication]], and [[Refinement and Fluency]].

Many other on-line lists of instructional and learning principles exist ['''''please add more!''''']:
* [http://ies.ed.gov/ncer/pubs/practiceguides/20072004.asp Organizing Instruction and Study to Improve Student Learning], one of the Practice Guides of the Department of Education, Institute for Education Sciences. See also:
** [http://www.nctq.org/dmsStage/Learning_About_Learning_Report A web-based update of this guide]
* [http://www.psyc.memphis.edu/learning/principles/ Principles of Learning] from Lifelong Learning at Work and at Home, a subgroup of the Association for Psychological Science.
* [http://www.edu-design-principles.org Design Principles Database] maintained by the NSF-funded [http://www.telscenter.org/ TELS (Technology Enhanced Learning in Science)] project
* [http://www.cmu.edu/teaching/principles/ Principles of Teaching and Learning] from CMU's Eberly Center for Teaching and Learning
* [http://en.wikipedia.org/wiki/Principles_of_learning Wikipedia entry for principles of learning]
* [http://www.udlcenter.org/aboutudl/udlguidelines Universal Design for Learning guidelines]
* See a review of principles and the role of technology in refining them in [https://www.youtube.com/watch?v=ysWTEWx9L_A this talk by Art Graesser].
* [https://kaitlinmctigue.github.io/interactive-principles/#/ An interactive website] describes the principles from our [https://science.sciencemag.org/content/342/6161/935 instructional complexity] paper which includes 30 principles merged from lists above: [[File:Instructional_Principles_Table.xlsx]].

[http://pact.cs.cmu.edu/koedinger/Koedinger10_WMV%20V9.wmv Ken Koedinger HCII Seminar Talk: Why Designing Effective Learning Interactions Is Not Easy and How We Can Do Better: Part I]

=== Toward a Hierarchical Structure of Robust Learning Hypotheses and Findings ===

The structure of the research cluster and study pages is as follows (a template can be found at [[Project Page Template and Creation Instructions]]). When a set of explanations share many terms and hypotheses, we make a node for each explanation, make a node for their common features, and link the nodes so that the common-feature node is the parent of each explanation node. In most cases a "node" is a single wiki page, but sometimes a node involves several wiki pages.

In order to more clearly display the integration, each node contains:
#An ''abstract'' that briefly describes the research encompassed by the node;
#A ''glossary'' that defines terms used elsewhere in this node but not defined in the nodes that are parents, grandparents, etc. of this node;
#The ''research question'' stated as concisely as possible, usually in a single sentence;
#A ''background and significance'' section that briefly summarizes prior work on the research question and why it is important to answer it;
#The ''dependent variables'', which are observable and typically measure competence, motivation, interaction, meta-learning, or some other pedagogically desirable outcome;
#The ''independent variables'', which typically include instructional environment, activity or method (the instructional "treatment" vs. "control"), and perhaps some student individual difference variables, such as gender or first language;
#The ''hypothesis'', which is a concise statement of the relationship among the variables that answers the research question;
#The ''findings'', which are the results of the study if it has been performed or the expected findings from the study if it has not -- explicitly indicate if the findings are preliminary;
#An ''explanation'' that describes the theoretical rationale for the hypothesis using the PSLC theoretical framework. It should be a paragraph or two and provide a causal chain that mentions mediating variables -- unobservable, hypothetical attributes of the students (e.g., knowledge components or path choices), how the treatment affects these, and how they, in turn, affect the dependent variables;
#The ''descendants'', which lists links to descendant nodes of this one, if there are any;
#A ''further information'' section that points to documents using hyper links and/or references in APA format. Each indicates briefly the document's relationship to the node (e.g., whether the document is a paper reporting the node in full detail, a proposal describing the motivation and design of the study in more detail, the node for a similar PSLC research study, etc.).

Experience suggests that the glossaries carry much of the load in explaining the research, and that carefully defining and exemplifying terms often pays off later in reducing confusion and facilitating collaboration. Consequently, the glossaries are sometimes so long that they are spit off as separate wiki pages.

The [[Root_node|root node of the hierarchy]] represents the overarching research question of how to achieve robust learning particularly in academic settings, like K12 schools and colleges. The question is necessarily abstract and is not the sort of question that can actually be tested by a single decisive experiment.

The immediate descendants of the root node are three nodes representing somewhat more specific research questions. There are nodes for each of [[Coordinative Learning]], [[Interactive Communication]] and [[Refinement and Fluency]]. These present somewhat more concrete research questions. They are specializations to the overarching questions, and form a bridge to testable hypotheses posed by individual research projects.

The leaves of the hierarchy (i.e., nodes with no descendants) represent individual research studies. A leaf node can also represent a group of studies or a whole project if the activities are sufficiently similar that it makes sense to summarize them with a single node. Each leaf node is maintained by its project’s leader and may or may not be publicly accessible depending on the state of the research.

Between the cluster nodes and the leaves, there may be some intervening nodes. For instance, if a group of Coordinative Learning studies all address a similar research question (e.g., how to use verbal and visual instruction together effectively), then a node may be created to summarize their shared aspects. Its parent is the Coordinative Learning cluster node, and its descendants are the relevant project nodes.

For a draft of [[Macro-level framework|an alternative framework based on independent dimensions of instruction click here]].

File:Instructional Principles Table.xlsx

2019-09-04T14:40:55Z

Koedinger: Koedinger uploaded a new version of File:Instructional Principles Table.xlsx

File:Instructional Principles Table.xlsx

2019-09-04T14:38:35Z

Koedinger:

Main Page

2019-09-04T14:27:34Z

Koedinger: /* Robust Learning Theoretical Framework */

== Robust Learning Theoretical Framework ==

A key goal of LearnLab is to support learning scientists in providing explanations of results using, as much as possible, the same core terminology and addressing an accumulating body of precise theoretical [[Instructional Principles and Hypotheses|principles of instruction]]. While a single theory of learning may emerge in the long term, the immediate goal is to encourage researchers to maximize the overlap between each others' theories. We want to help the field get beyond the "[[Toothbrush Problem]]" in theorizing.

In 2012, we published a theoretical framework, called the [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Knowledge-Learning-Instruction (KLI) framework]. This framework builds on our 2006 [http://learnlab.org/clusters/PSLC_Theory_Frame_June_15_2006.pdf theoretical framework document] and, importantly, on the contributions to this wiki. In 2013, we published [http://pact.cs.cmu.edu/pubs/Koedinger-Science-2013.pdf a paper in Science] reviewing instructional principles, the complexity of their combinations, and recommendations for LearnLab style research to address this complexity.

In order to display the integration across research projects, this wiki maintains multiple theoretical hierarchies, one of [[Instructional Principles and Hypotheses|Instructional Principles]] and another of empirical studies, which are found on the cluster pages: [[Coordinative Learning]], [[Interactive Communication]], and [[Refinement and Fluency]].

Many other on-line lists of instructional and learning principles exist ['''''please add more!''''']:
* [http://ies.ed.gov/ncer/pubs/practiceguides/20072004.asp Organizing Instruction and Study to Improve Student Learning], one of the Practice Guides of the Department of Education, Institute for Education Sciences. See also:
** [http://www.nctq.org/dmsStage/Learning_About_Learning_Report A web-based update of this guide]
* [http://www.psyc.memphis.edu/learning/principles/ Principles of Learning] from Lifelong Learning at Work and at Home, a subgroup of the Association for Psychological Science.
* [http://www.edu-design-principles.org Design Principles Database] maintained by the NSF-funded [http://www.telscenter.org/ TELS (Technology Enhanced Learning in Science)] project
* [http://www.cmu.edu/teaching/principles/ Principles of Teaching and Learning] from CMU's Eberly Center for Teaching and Learning
* [http://en.wikipedia.org/wiki/Principles_of_learning Wikipedia entry for principles of learning]
* [http://www.udlcenter.org/aboutudl/udlguidelines Universal Design for Learning guidelines]
* See a review of principles and the role of technology in refining them in [https://www.youtube.com/watch?v=ysWTEWx9L_A this talk by Art Graesser].
* [https://kaitlinmctigue.github.io/interactive-principles/#/ An interactive website] describes the principles from our [https://science.sciencemag.org/content/342/6161/935 instructional complexity] paper which includes [https://science.sciencemag.org/content/suppl/2013/11/20/342.6161.935.DC1?_ga=2.150241526.1122111306.1567539400-1203323450.1562771465 30 principles] merged from lists above.

[http://pact.cs.cmu.edu/koedinger/Koedinger10_WMV%20V9.wmv Ken Koedinger HCII Seminar Talk: Why Designing Effective Learning Interactions Is Not Easy and How We Can Do Better: Part I]

=== Toward a Hierarchical Structure of Robust Learning Hypotheses and Findings ===

The structure of the research cluster and study pages is as follows (a template can be found at [[Project Page Template and Creation Instructions]]). When a set of explanations share many terms and hypotheses, we make a node for each explanation, make a node for their common features, and link the nodes so that the common-feature node is the parent of each explanation node. In most cases a "node" is a single wiki page, but sometimes a node involves several wiki pages.

In order to more clearly display the integration, each node contains:
#An ''abstract'' that briefly describes the research encompassed by the node;
#A ''glossary'' that defines terms used elsewhere in this node but not defined in the nodes that are parents, grandparents, etc. of this node;
#The ''research question'' stated as concisely as possible, usually in a single sentence;
#A ''background and significance'' section that briefly summarizes prior work on the research question and why it is important to answer it;
#The ''dependent variables'', which are observable and typically measure competence, motivation, interaction, meta-learning, or some other pedagogically desirable outcome;
#The ''independent variables'', which typically include instructional environment, activity or method (the instructional "treatment" vs. "control"), and perhaps some student individual difference variables, such as gender or first language;
#The ''hypothesis'', which is a concise statement of the relationship among the variables that answers the research question;
#The ''findings'', which are the results of the study if it has been performed or the expected findings from the study if it has not -- explicitly indicate if the findings are preliminary;
#An ''explanation'' that describes the theoretical rationale for the hypothesis using the PSLC theoretical framework. It should be a paragraph or two and provide a causal chain that mentions mediating variables -- unobservable, hypothetical attributes of the students (e.g., knowledge components or path choices), how the treatment affects these, and how they, in turn, affect the dependent variables;
#The ''descendants'', which lists links to descendant nodes of this one, if there are any;
#A ''further information'' section that points to documents using hyper links and/or references in APA format. Each indicates briefly the document's relationship to the node (e.g., whether the document is a paper reporting the node in full detail, a proposal describing the motivation and design of the study in more detail, the node for a similar PSLC research study, etc.).

Experience suggests that the glossaries carry much of the load in explaining the research, and that carefully defining and exemplifying terms often pays off later in reducing confusion and facilitating collaboration. Consequently, the glossaries are sometimes so long that they are spit off as separate wiki pages.

The [[Root_node|root node of the hierarchy]] represents the overarching research question of how to achieve robust learning particularly in academic settings, like K12 schools and colleges. The question is necessarily abstract and is not the sort of question that can actually be tested by a single decisive experiment.

The immediate descendants of the root node are three nodes representing somewhat more specific research questions. There are nodes for each of [[Coordinative Learning]], [[Interactive Communication]] and [[Refinement and Fluency]]. These present somewhat more concrete research questions. They are specializations to the overarching questions, and form a bridge to testable hypotheses posed by individual research projects.

The leaves of the hierarchy (i.e., nodes with no descendants) represent individual research studies. A leaf node can also represent a group of studies or a whole project if the activities are sufficiently similar that it makes sense to summarize them with a single node. Each leaf node is maintained by its project’s leader and may or may not be publicly accessible depending on the state of the research.

Between the cluster nodes and the leaves, there may be some intervening nodes. For instance, if a group of Coordinative Learning studies all address a similar research question (e.g., how to use verbal and visual instruction together effectively), then a node may be created to summarize their shared aspects. Its parent is the Coordinative Learning cluster node, and its descendants are the relevant project nodes.

For a draft of [[Macro-level framework|an alternative framework based on independent dimensions of instruction click here]].

E-Learning Design Principles and Methods 2017

2017-07-26T18:34:42Z

Koedinger: Created page with "====Course Details==== Course number: 05-823 Semester: Fall 2017 Carnegie Mellon University =====Class times===== 9:00 to 10:20 Tuesday & Thursday =====Location===== Gat..."

====Course Details====
Course number: 05-823

Semester: Fall 2017

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Administrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 4. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-14
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 8-31"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?
*** Do posts for the readings.

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''Midterm review''
**Do review quiz and bring questions to class
**No new post or quiz.
**Optional Readings:
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''

=====Multimedia Principles 10-27 to 11-8=====
*'''10-27''' ''Optional topic''
**E-Learning in Industry
**Work on project
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-3''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-8''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-10 to 11-24=====

*'''11-10''' ''Applying the Guidelines; KLI & Selecting appropriate instructional principles''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***Do posts for the reading.
** Time permitting: Peer review of instructional design

*'''11-15''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-17''' ''Finish discussion of experimentation''

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2017-07-26T18:32:54Z

Koedinger:

A new version of this course is at [[E-Learning Design Principles and Methods 2017]]

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 4. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-14
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 8-31"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?
*** Do posts for the readings.

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''Midterm review''
**Do review quiz and bring questions to class
**No new post or quiz.
**Optional Readings:
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''

=====Multimedia Principles 10-27 to 11-8=====
*'''10-27''' ''Optional topic''
**E-Learning in Industry
**Work on project
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-3''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-8''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-10 to 11-24=====

*'''11-10''' ''Applying the Guidelines; KLI & Selecting appropriate instructional principles''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***Do posts for the reading.
** Time permitting: Peer review of instructional design

*'''11-15''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-17''' ''Finish discussion of experimentation''

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

Worked example principle

2017-07-12T15:52:29Z

Koedinger: /* Laboratory experiment support */

== Brief statement of principle ==

In contrast to the traditional approach of giving a list homework (or seatwork) problems for students to solve, students learn more efficiently and more robustly when more frequent study of worked examples is interleaved with problem solving practice.

== Description of principle ==

"In courses that are teaching new tasks, learning time can be saved by replacing some practice problems with worked examples" (Clark & Mayer, 2004, p. 177). In addition, most studies comparing interleaved worked examples and problems with all problems have also shown improved learning outcomes, including robust learning outcomes.

"It would be an unusual (not to mention incompetent) teacher who did not use worked examples.  Similarly, textbooks universally use worked examples to illustrate new concepts.  The suggestion being made here goes beyond this limited use of worked examples.  Rather than using them merely to demonstrate how to use a mathematical or scientific rule, the proposal is that they should be used in large numbers as a form of practice.  In other words, instead of practicing by solving many problems (an activity engaged in by most conscientious students), it is proposed that many of these problems could profitably be replaced by worked examples." Sweller, J. (1999) p73

=== Operational definition ===

=== Examples ===

Imagine instead of giving students a typical homework or seatwork assignment involving 8 problems, you give them an assignment where every other problem comes with a complete worked out solution. The even numbered items would be usual problems, like the following algebra problem:
Solve 12 + 2x = 15 for x

The odd numbered problems, come with solutions, like this:
Solve 12 + 2x = 15 for x
Study each step in this solution, so that you can better solve the next problem on your own:
12+2x = 15
2x = 15-12
2x = 3
x = 3/2
x = 1.5

Which approach, asking for solutions to all 8 problems or interleaving 4 examples with 4 problems, will lead to better student learning? You might think that the 8 problems require more work or that students might ignore the examples and thus, the 8 problems would lead to more learning. But, much research has shown that students typically learn more deeply and more easily from the second approach, when examples are interleaved between problems.

Teachers often think so many examples “give it away” or that students will not pay attention to the example. But, by having problems in between students are motivated to pay more attention to the example so as to prepare for the next problem or to resolve a question from the past problem. The problems break a students’ “illusion of knowing” that might otherwise lead them to skim the example and believe it is obvious.

It is important that students spend time actively engaged in learning and in genuine problem solving and reasoning. However, an emphasis on “learn by doing” is sometimes taken too far and students end up with homework problems or projects that are beyond their means. In such cases, they may spend much unproductive study time struggling without success. This time is often not only wasted but may increase a students’ frustration with the subject-matter and lead to unjustified feelings of not being good at math or science particularly. In contrast, during example study, students can focus their attention on understanding the principles underlying the examples instead of simply on finishing the problem. In early learning, the thought that goes simply into trying to solve the problem seems to distract students from trying to understand the principles underlying the solution.

Notice that in the example above, explanations for each step are not provided. It is best when students provide these explanations themselves (see the [[prompted self-explanation hypothesis]]) and, while more research is needed, providing explanations can sometimes distract students from doing so themselves and in other cases seems to provide no additional enhancement in student learning.

In whole classroom situation a teacher might implement this principle by going back and forth between a classroom or small group discussion around an example solution followed by small groups or individuals solving a problem (just one!) on their own. Then back to example study, for instance, by having students present their solutions and having others attempt to explain the steps (see the [[prompted self-explanation hypothesis]]). Now back to a second problem.

By giving the students frequent opportunities to study examples in between problem solving, students can more easily and more deeply acquire the big ideas, key concepts, or key principles that we want them to learn. With greater understanding, students will do better on harder problems in the future that require them to transfer these key concepts beyond the problems just like those they have seen before.

== Experimental support ==

As summarized in Clark & Mayer, 2003 (pp 179): "There is a lot of evidence for the effectiveness of learning from worked examples.  As an example, in one study twelve [statistics] problems were used.  In the conventional group the learners solved all twelve problems as practice.  In the worked examples group, the learners received eight problems already worked out to study and then four problems to solve as practice.  Students in the worked examples group spent significantly less time studying and scored higher on a test than did those in the conventional group.  Furthermore, the worked examples group scored higher not only on test problems similar to those used during practice but also on different types of problems requiring application of the principles taught (Paas, 1992).  The investigators conclude that 'training with partly or completely worked-out problems leads to less effort-demanding and better transfer performance and is more time efficient' (p. 433).  In fact, in one study, the use of worked examples allowed learners to complete a three-year mathematics course in two years (Zhu and Simon, 1987).  Positive effects of worked examples have been reported in a variety of courses teaching well-defined problems, including algebra, geometry, statistics, and programming". 

=== Laboratory experiment support ===

See papers Cooper & Sweller and many others, such as Atkinson, Ayres, Catrambone, Paas, Renkl, van Gog, van Merrienboer ... See the recommendation "Repeatedly alternating problems with their solutions provided and problems that students must solve" [http://www.nctq.org/dmsStage/Learning_About_Learning_Report on this web page] which is a web-based update of this IES practice guide: Organizing Instruction and Study to Improve Student Learning. Some relevant references can be found within [http://www.nctq.org/dmsView/App_G this page.]

=== In vivo experiment support ===

[[McLaren_et_al_-_Studying_the_Learning_Effect_of_Personalization_and_Worked_Examples_in_the_Solving_of_Stoich_Problems | McLaren's three stoichiometry studies]] provide mixed support of the worked example principle. Although ''students did not learn more'' through the study of worked examples followed by problem solving, as in (Paas, 1992; Zhu and Simon, 1987; Trafton & Reiser, 1993), ''they did learn more efficiently'' as in the earlier studies. On the other hand, only normal pre-post gains were evaluated in the stoichiometry studies; [[robust learning]] was not measured. In addition, the control condition of these three studies was different -- and potentially much more rigorous -- than the earlier studies: students solved problems with the support of an ''intelligent tutor''. This may explain why students did not learn more: perhaps the additional support of the tutor -- in which students theoretically could create their own "worked examples" by clicking through to bottom out hints -- equalizes the advantage of learning from the examples.

[[Does_learning_from_worked-out_examples_improve_tutored_problem_solving? | Salden, Renkl, Schwonke and Aleven ]] (2008) studied the use of 'faded' examples as an adjunct to tutored problem solving with the Geometry Cognitive Tutor. Following Renkl and Atkinson's (2003) example fading methods, the example-enhanced version of the tutor, after first presenting fully-worked-out examples, gradually reduced the number of solution steps given, thus increasing the number of open steps that students had to solve. Salden et al. found that the faded examples help students learn more efficiently and effectively (e.g., Schwonke, Renkl, Krieg, Wittwer, Aleven, & Salden, in press) especially when the examples are faded in an adaptive manner, responsive to the students' explanations of worked-out steps.

Paas (1992): Across 12 tasks, interleaving 2 worked examples and 1 problem to solve leads to:
* shorter learning time-on-task than solving all active problem solving (computing mean, median, mode)
* similar time on task and accuracy for solving the 4 active problem solving items
* better near and far transfer for worked examples vs. active problem solving
* greater perceived mental effort for active problem solving than worked examples

== Theoretical rationale ==

The original rational for the worked example effect came from Sweller's Cognitive Load Theory (Sweller, 1988; Sweller & Cooper, 1985):

"[[Working memory]] has a limited capacity that becomes inefficient when having to retain even a few items. If the only way to build job-relevant skills is to perform many practice exercises, working memory can become overloaded by the mental work required to complete these exercises. However, if limited working memory resources could be used to study worked examples and build new knowledge from them, some of this labor-intensive effort could be bypassed. Worked examples are more efficient for learning new tasks because they reduce the load in working memory, thereby allowing the learner to learn the steps in problem solving. Sweller and his colleagues distinguished between the intrinsic load of instructional materials that result from the inherent complexity of the content itself and the extraneous load imposed by the instructional design (Sweller, 1999; Sweller, Van Merrienboer and Paas, 1998). Learners who are studying complex topics will have to deal with high intrinsic mental load, especially if it's new information. However, good e-learning can help learners manage that lead by using effective instructional methods. Replacing some assigned problems with worked examples reduces the extraneous load, freeing working memory to allocate resources to the learning process. This recommendation applies primarily to courses for novice learners who are most susceptible to cognitive overload". (Clark & Mayer, 2003, pp. 178-179)

Another line of rationale suggests that worked examples make students engage in more [[self-explanation]] than they do during problem solving.

One (of perhaps many) open questions is what motivates students to process examples more deeply, that is, to engage in "generative processing" (Mayer) or "germane load" (Van Merrienboer and Paas?). The importance of interleaving examples and problems may be primarily about motivating students to deeply process the examples. Such an explanation is different from the "knowledge compilation" explanation for interleaving articulated by Trafton & Reiser (1993).

A related line of reasoning suggests that example study better engages explicit learning (based on verbal rules or principles communicated in instruction) than does problem solving practice. By engaging in explicit reasoning about the domain rules or principles, students are more likely to discriminate relevant from irrelevant features of those rules, that is, more likely to engage in explicit [[refinement]]. In contrast, problem solving drives attentive example study. It breaks students "illusion of knowing" and motivates more careful example study (as mentioned above). Problem solving appears important for turning slow explicit processing into fast habit-like processing. This role of problem solving is what Trafton & Reiser called "knowledge compilation", which is further elaborated in Anderson's ACT-R theory (Anderson, Fincham, & Douglass, 1997). Combining example study and problem solving thus draws on their complementary benefits. By interleaving the two, example study remains more focused and problem solving is more likely to "stamp in" accurate [[knowledge components]] that employ the relevant retrieval features and avoid irrelevant ones (i.e., have high [[feature validity]]).

== Conditions of application ==

''1. Interleave examples and problems''. Trafton & Reiser (1993) showed that examples and problems should be given in an alternating or interleaved order (Example, Problem, Example, Problem, ...) and not blocked (Example, Example, ..., Problem, Problem, ...). This was the approach taken in [[McLaren_et_al_-_Studying_the_Learning_Effect_of_Personalization_and_Worked_Examples_in_the_Solving_of_Stoich_Problems | McLaren's PSLC studies]].

''2. Switch to problems later in learning. ''The "expertise-reversal effect" suggests that it is earlier in skill development when the Worked Example Principle will be applicable, whereas later in development have students just solve problems without interleaved examples may be better (Kalyuga, Chandler, Tuovinen, & Sweller, 2001).

''3. Including explanations in examples helps when there are no self-explanation prompts, but hurts when there are self-explanation prompts''.  See the discussion of not providing explanations in the example above in the Examples section.  Schworm and Renkl have explored this issue contrasting whether <i>instructional</i> explanations (given on demand) are present or not, and (in a second study) whether self-explanation prompts are present or not (ADD REFS to Renkl).  

''4. Indicate subgoals in the example''.  In constrast to null or negative effects of adding explanations to examples (i.e., statements that justify a step), indicating how the steps fit into a hierarchy of goals and subgoals (e.g., by labeling some steps as key subgoals) does appear to aid learning.  (ADD REFS to Catrambone).

''5. Separate example study from problem solving''.  Having the example present during problem solving may encourage shallow processing (i.e., copying and small edits without understanding) of the example and may not yield benefit.   While there is clear theoretical support for this condition of application, there does not seem to be more solid experimental evidence for it.  Preliminary results from [[In vivo comparison of Cognitive Tutor Algebra using handwriting vs typing input|Anthony's PSLC study]] are consistent with the idea that the worked example effect is not found when examples are provided to students while they are asked to solve an analogous problem.<br>

''6. Tell students to study the example to prepare for upcoming problem solving''.  According to John Sweller (personal communication with Ken Koedinger), in his experiments, students were instructed at the beginning to study each example in preparation for upcoming problem solving.  The prompting is recommended as critical to give students motivation to attend to and study the example.  It is not clear whether there is any experimental support for this condition of application (i.e., comparing learning with this instruction vs. without). [Need to add references, this may be described in Sweller's book, Sweller, 1999] Note that, unlike the Sweller studies, [[McLaren_et_al_-_Studying_the_Learning_Effect_of_Personalization_and_Worked_Examples_in_the_Solving_of_Stoich_Problems | McLaren's PSLC studies]] did not instruct students to study worked examples in preparation for problem solving. Rather, students were simply presented worked examples, with no preparation. These studies resulted in a worked example benefit, with respect to efficiency but not with respect to learning (at least standard pre-post learning).

==Caveats, limitations, open issues, or dissenting views==
==Variations (descendants)==
== Generalizations (ascendants) ==

[[Example-rule coordination principle]]

== References ==

* Anderson, J. R., Fincham, J. M., & Douglass, S. (1997). The role of examples and rules in the acquisition of cognitive skill. Journal of Experimental Psychology: Learning Memory, and Cognition, 23(4), 932–945.

* Atkinson, R., Derry, S.J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), 181-214.

* Clark, R. C., & Mayer, R. E. (2003). e-Learning and the Science of Instruction : Proven Guidelines for Consumers and Designers of Multimedia Learning. San Francisco: Jossey-Bass.

* Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93(3), 579–588.

* Lovett, M.C. (1992). Learning by problem solving versus by examples: The benefits of generating and receiving information. In: 14th Annual Conference of the Cognitive Science Society, pp. 956-961. Hillsdale, NJ: Erlbaum.

* Paas, F. (1992). Training strategies for attaining transfer of problem solving skill in statistics: A cognitive load approach. Journal of Educational Psychology, 84, 429–434.

* Renkl, A. & Atkinson, R. K. (2003). Structuring the transition from example study to problem solving in cognitive skills acquisition: A cognitive load perspective. Educational Psychologist, 38, 15-22.

* Salden, R., Aleven, V., Renkl, A., & Schwonke, R. (2008). Worked examples and tutored problem solving: redundant or synergistic forms of support? In C. Schunn (Ed.) Proceedings of the Annual Meeting of the Cognitive Science Society, CogSci 2008. New York, NY: Lawrence Erlbaum. Cognition and Student Learning Prize.

* Schwonke, R., Renkl, A., Krieg, C., Wittwer, J., Aleven, V., & Salden, R. (in press). The worked-example effect: is it just an artefact of lousy control conditions? Computers in Human Behavior.

* Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2, 59–89.

* Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285.

* Sweller, J. (1999). Instructional design in technical areas.  Camberwell, Australia: ACER Press

* Sweller, J., van Merrienboer, J.J.G., & Paas, F. (1998).  Cognitive architecture and instructional design.  Educational Psychology Review, 10, 251-296

* Trafton, J. G., & Reiser, B. J. (1993). The contribution of studying examples and solving problems to skill acquisition. Proceedings of the 15th Annual Conference of the Cognitive Science Society (pp. 1017–1022). Hillsdale: Lawrence Erlbaum Associates, Inc.

* Zhu, X., & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166.

[[Category:Glossary]]
[[Category:Instructional Principle]]

Main Page

2017-07-12T15:31:09Z

Koedinger: /* Robust Learning Theoretical Framework */ Removed link to web page that no longer exists: ** [http://dww.ed.gov/topic/topic_landing.cfm?PA_ID=9&T_ID=19&Tab=1 The web page "Doing What Works: Psychology of Learning"]

== Robust Learning Theoretical Framework ==

A key goal of LearnLab is to support learning scientists in providing explanations of results using, as much as possible, the same core terminology and addressing an accumulating body of precise theoretical [[Instructional Principles and Hypotheses|principles of instruction]]. While a single theory of learning may emerge in the long term, the immediate goal is to encourage researchers to maximize the overlap between each others' theories. We want to help the field get beyond the "[[Toothbrush Problem]]" in theorizing.

In 2012, we published a theoretical framework, called the [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Knowledge-Learning-Instruction (KLI) framework]. This framework builds on our 2006 [http://learnlab.org/clusters/PSLC_Theory_Frame_June_15_2006.pdf theoretical framework document] and, importantly, on the contributions to this wiki. In 2013, we published [http://pact.cs.cmu.edu/pubs/Koedinger-Science-2013.pdf a paper in Science] reviewing instructional principles, the complexity of their combinations, and recommendations for LearnLab style research to address this complexity.

In order to display the integration across research projects, this wiki maintains multiple theoretical hierarchies, one of [[Instructional Principles and Hypotheses|Instructional Principles]] and another of empirical studies, which are found on the cluster pages: [[Coordinative Learning]], [[Interactive Communication]], and [[Refinement and Fluency]].

Many other on-line lists of instructional and learning principles exist ['''''please add more!''''']:
* [http://ies.ed.gov/ncer/pubs/practiceguides/20072004.asp Organizing Instruction and Study to Improve Student Learning], one of the Practice Guides of the Department of Education, Institute for Education Sciences. See also:
** [http://www.nctq.org/dmsStage/Learning_About_Learning_Report A web-based update of this guide]
* [http://www.psyc.memphis.edu/learning/principles/ Principles of Learning] from Lifelong Learning at Work and at Home, a subgroup of the Association for Psychological Science.
* [http://www.edu-design-principles.org Design Principles Database] maintained by the NSF-funded [http://www.telscenter.org/ TELS (Technology Enhanced Learning in Science)] project
* [http://www.cmu.edu/teaching/principles/ Principles of Teaching and Learning] from CMU's Eberly Center for Teaching and Learning
* [http://en.wikipedia.org/wiki/Principles_of_learning Wikipedia entry for principles of learning]
* [http://www.udlcenter.org/aboutudl/udlguidelines Universal Design for Learning guidelines]
* See a review of principles and the role of technology in refining them in [https://www.youtube.com/watch?v=ysWTEWx9L_A this talk by Art Graesser].

[http://pact.cs.cmu.edu/koedinger/Koedinger10_WMV%20V9.wmv Ken Koedinger HCII Seminar Talk: Why Designing Effective Learning Interactions Is Not Easy and How We Can Do Better: Part I]

=== Toward a Hierarchical Structure of Robust Learning Hypotheses and Findings ===

The structure of the research cluster and study pages is as follows (a template can be found at [[Project Page Template and Creation Instructions]]). When a set of explanations share many terms and hypotheses, we make a node for each explanation, make a node for their common features, and link the nodes so that the common-feature node is the parent of each explanation node. In most cases a "node" is a single wiki page, but sometimes a node involves several wiki pages.

In order to more clearly display the integration, each node contains:
#An ''abstract'' that briefly describes the research encompassed by the node;
#A ''glossary'' that defines terms used elsewhere in this node but not defined in the nodes that are parents, grandparents, etc. of this node;
#The ''research question'' stated as concisely as possible, usually in a single sentence;
#A ''background and significance'' section that briefly summarizes prior work on the research question and why it is important to answer it;
#The ''dependent variables'', which are observable and typically measure competence, motivation, interaction, meta-learning, or some other pedagogically desirable outcome;
#The ''independent variables'', which typically include instructional environment, activity or method (the instructional "treatment" vs. "control"), and perhaps some student individual difference variables, such as gender or first language;
#The ''hypothesis'', which is a concise statement of the relationship among the variables that answers the research question;
#The ''findings'', which are the results of the study if it has been performed or the expected findings from the study if it has not -- explicitly indicate if the findings are preliminary;
#An ''explanation'' that describes the theoretical rationale for the hypothesis using the PSLC theoretical framework. It should be a paragraph or two and provide a causal chain that mentions mediating variables -- unobservable, hypothetical attributes of the students (e.g., knowledge components or path choices), how the treatment affects these, and how they, in turn, affect the dependent variables;
#The ''descendants'', which lists links to descendant nodes of this one, if there are any;
#A ''further information'' section that points to documents using hyper links and/or references in APA format. Each indicates briefly the document's relationship to the node (e.g., whether the document is a paper reporting the node in full detail, a proposal describing the motivation and design of the study in more detail, the node for a similar PSLC research study, etc.).

Experience suggests that the glossaries carry much of the load in explaining the research, and that carefully defining and exemplifying terms often pays off later in reducing confusion and facilitating collaboration. Consequently, the glossaries are sometimes so long that they are spit off as separate wiki pages.

The [[Root_node|root node of the hierarchy]] represents the overarching research question of how to achieve robust learning particularly in academic settings, like K12 schools and colleges. The question is necessarily abstract and is not the sort of question that can actually be tested by a single decisive experiment.

The immediate descendants of the root node are three nodes representing somewhat more specific research questions. There are nodes for each of [[Coordinative Learning]], [[Interactive Communication]] and [[Refinement and Fluency]]. These present somewhat more concrete research questions. They are specializations to the overarching questions, and form a bridge to testable hypotheses posed by individual research projects.

The leaves of the hierarchy (i.e., nodes with no descendants) represent individual research studies. A leaf node can also represent a group of studies or a whole project if the activities are sufficiently similar that it makes sense to summarize them with a single node. Each leaf node is maintained by its project’s leader and may or may not be publicly accessible depending on the state of the research.

Between the cluster nodes and the leaves, there may be some intervening nodes. For instance, if a group of Coordinative Learning studies all address a similar research question (e.g., how to use verbal and visual instruction together effectively), then a node may be created to summarize their shared aspects. Its parent is the Coordinative Learning cluster node, and its descendants are the relevant project nodes.

For a draft of [[Macro-level framework|an alternative framework based on independent dimensions of instruction click here]].

Educational Research Methods 2017

2017-04-23T15:47:35Z

Koedinger: /* Experimental Research Methods (Koedinger) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
** Guest Instructor: Rebecca Nugent
* Cognitive Task Analysis - Quantitative: Mar 9 (T)
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Amy Ogan
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day (Educational Design Research?): Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, Reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-23

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-2

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-7 Continued discussion of Psychometrics

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) comfortably before class on Thursday -- by 3pm. Also, in addition to the problem content file indicated in the assignment handout see other files in the same location to get a more complete description and list of the files: Geometry Area Problems PDF Explanation.docx and solutions.zip.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (at least get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-5.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified? (Look for datasets with the lego block icon on them -- these have associated problem descriptions)
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions on any of the metrics, AIC, BIC, or cross validation?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate AFM in R using either glm or glmer (in package lme4). You
can find R code that mimics AFM in the DataShop help, here:
https://pslcdatashop.web.cmu.edu/help?page=rSoftware

How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (e.g., as measured by AIC or BIC or cross validation)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25
**First thing: Do "Experimental Methods" Quiz on Blackboard
**Make progress on your project -- come prepared to tell us about it!
**Reading: Start Trochim's Ch 7 and 9
**Optional: Try ANOVA module of OLI Statistics course
**Relevant Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*4-27
**Reading: Finish Trochim's Ch 7 and 9
**Optional: Try ANOVA module of OLI Statistics course
**Do two posts on Blackboard.
*5-2
**Reading: Trochim Ch 10
**Relevant Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-03-30T12:09:58Z

Koedinger: /* Educational Data Mining -- Learning Curve Analysis (Koedinger) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
** Guest Instructor: Rebecca Nugent
* Cognitive Task Analysis - Quantitative: Mar 9 (T)
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Amy Ogan
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day (Educational Design Research?): Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, Reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-23

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-2

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-7 Continued discussion of Psychometrics

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) comfortably before class on Thursday -- by 3pm. Also, in addition to the problem content file indicated in the assignment handout see other files in the same location to get a more complete description and list of the files: Geometry Area Problems PDF Explanation.docx and solutions.zip.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (at least get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-5.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified? (Look for datasets with the lego block icon on them -- these have associated problem descriptions)
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions on any of the metrics, AIC, BIC, or cross validation?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate AFM in R using either glm or glmer (in package lme4). You
can find R code that mimics AFM in the DataShop help, here:
https://pslcdatashop.web.cmu.edu/help?page=rSoftware

How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (e.g., as measured by AIC or BIC or cross validation)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-03-29T20:54:45Z

Koedinger: /* Educational Data Mining -- Learning Curve Analysis (Koedinger) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
** Guest Instructor: Rebecca Nugent
* Cognitive Task Analysis - Quantitative: Mar 9 (T)
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Amy Ogan
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day (Educational Design Research?): Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, Reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-23

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-2

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-7 Continued discussion of Psychometrics

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) comfortably before class on Thursday -- by 3pm. Also, in addition to the problem content file indicated in the assignment handout see other files in the same location to get a more complete description and list of the files: Geometry Area Problems PDF Explanation.docx and solutions.zip.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (at least get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-5.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate AFM in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-03-29T15:11:53Z

Koedinger: /* Educational Data Mining -- Learning Curve Analysis (Koedinger) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
** Guest Instructor: Rebecca Nugent
* Cognitive Task Analysis - Quantitative: Mar 9 (T)
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Amy Ogan
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day (Educational Design Research?): Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, Reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-23

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-2

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-7 Continued discussion of Psychometrics

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) comfortably before class on Thursday -- by 3pm. Also, in addition to the problem content file indicated in the assignment handout see other files in the same location to get a more complete description and list of the files: Geometry Area Problems PDF Explanation.docx and solutions.zip.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-03-22T20:25:46Z

Koedinger: /* Surveys, Questionnaires, Interviews (Ogan) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
** Guest Instructor: Rebecca Nugent
* Cognitive Task Analysis - Quantitative: Mar 9 (T)
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Amy Ogan
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day (Educational Design Research?): Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, Reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-23

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-2

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-7 Continued discussion of Psychometrics

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-02-14T23:43:41Z

Koedinger: /* Psychometrics, reliability, Item Response Theory (Nugent) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
** Guest Instructor: Rebecca Nugent
* Cognitive Task Analysis - Quantitative: Mar 9 (T)
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Amy Ogan
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day (Educational Design Research?): Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, Reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-23

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-28

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-2

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-7 Continued discussion of Psychometrics

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-02-14T23:42:15Z

Koedinger: /* Cognitive Task Analysis (CTA) (Koedinger) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
** Guest Instructor: Rebecca Nugent
* Cognitive Task Analysis - Quantitative: Mar 9 (T)
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Amy Ogan
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day (Educational Design Research?): Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*3-9[!NOT IN ORDER!] Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-02-14T23:37:12Z

Koedinger: /* Class Schedule in Brief */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21 (TRT)
* Educational Measurement & Psychometrics: Feb 23, 28, Mar 2, 7 (RTRT)
** Guest Instructor: Rebecca Nugent
* Cognitive Task Analysis - Quantitative: Mar 9 (T)
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Amy Ogan
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day (Educational Design Research?): Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-02-06T16:44:57Z

Koedinger: /* Psychometrics, reliability, Item Response Theory (Nugent) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Nugent)=====

* TO BE DETERMINED: Plans for these classes will communicated by Rebecca Nugent.

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-01-24T22:24:06Z

Koedinger: /* Video and Verbal Protocol Analysis (Lovett, Rosé) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The plan for this session and readings are in [[Media:2017 Verbal Data Analysis Unit.zip|this zip file]], which is also available on blackboard.

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

File:2017 Verbal Data Analysis Unit.zip

2017-01-24T22:21:33Z

Koedinger:

Educational Research Methods 2017

2017-01-17T21:20:51Z

Koedinger: /* Flipped Homework: Reading Reports and Pre-Class Assignments */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 3:30pm''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

[TO BE UPDATED]
The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 24]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 26 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 31 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 2 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 7 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 9 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-01-17T21:20:04Z

Koedinger: /* Textbook and Readings */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly.

Find it by googling for the title or [https://www.google.com/search?q=The+Research+Methods+Knowledge+Base%3A+3rd+edition&ie=utf-8&oe=utf-8 clicking here].

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

[TO BE UPDATED]
The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 24]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 26 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 31 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 2 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 7 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 9 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-01-17T17:05:09Z

Koedinger: /* Surveys, Questionnaires, Interviews (Kiesler) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

[TO BE UPDATED]
The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 24]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 26 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 31 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 2 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 7 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 9 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Ogan) =====
* [Plans for these classes may be communicated by Ogan (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-01-17T16:53:44Z

Koedinger: /* Instructor */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger
Office: 3601 Newell-Simon Hall, Phone: 412-268-7667
Email: Koedinger@cmu.edu, Office hours by appointment

Other instructors: Carolyn Rose, Marsha Lovett, Amy Ogan, Rebecca Nugent, Richard Scheines

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

[TO BE UPDATED]
The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 24]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 26 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 31 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 2 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 7 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 9 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
* [Plans for these classes were communicated by Kiesler (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-01-17T13:10:16Z

Koedinger: /* Location */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4101 Gates/Hillman

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

[TO BE UPDATED]
The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 24]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 26 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 31 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 2 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 7 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 9 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
* [Plans for these classes were communicated by Kiesler (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-01-15T14:49:15Z

Koedinger: /* Class Schedule with Readings and Assignments */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4301 Gates/Hillman

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

[TO BE UPDATED]
The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 24]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 26 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 31 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 2 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 7 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 9 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
* [Plans for these classes were communicated by Kiesler (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

Educational Research Methods 2017

2017-01-15T14:48:01Z

Koedinger: /* Course Intro, Research Questions, Picking Methods (Koedinger) */

===Research Methods for the Learning Sciences 05-748===
Spring 2017 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
4301 Gates/Hillman

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 17 (T)
* Choosing Qualitative & Quantitative Methods: Jan 19 (R)
* Video and Verbal Protocol Analysis: Jan 24, 26, 31, Feb 2, 7, 9 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 14, 16, 21, 23 (TRTR)
* Educational Design Research: Feb 28 (T)
* Educational Measurement & Psychometrics: Mar 2, 7, 9, (RTR)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 14, 16 (TR)
* Surveys, Questionnaires, Interviews: Mar 21, 23 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 28, 30, Apr 4 (TRT)
* Flex day: Apr 6 (R)
* Educational Data Mining & Causal Inference: Apr 11, 13, 18, (TRT)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 20 (R)
* Experimental Methods: Apr 25, 27, May 2, 4 (TRTR)
* Wrap-up: May 9 (T)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-17
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-19 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The 2017 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 24]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 26 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 31 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Feb 2 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 7 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 9 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-14 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-16 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-21 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-23 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-28

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*3-2

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-7

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-9 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-14 and 3-16 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
* [Plans for these classes were communicated by Kiesler (& Koedinger) via email.]
*3-21
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-23
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-28
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-30
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-4
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-6 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-11
**Before class on 4-11, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-13
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-18
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

*4-20 NO CLASS - Spring Carnival

=====Experimental Research Methods (Koedinger)=====
*4-25 Continuation of Causal Inference
*4-27
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*5-2
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-4
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 11.

E-Learning Design Principles and Methods 2016

2016-10-27T12:39:08Z

Koedinger: /* Multimedia Principles 10-27 to 11-8 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?
*** Do posts for the readings.

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''Midterm review''
**Do review quiz and bring questions to class
**No new post or quiz.
**Optional Readings:
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''

=====Multimedia Principles 10-27 to 11-8=====
*'''10-27''' ''Optional topic''
**E-Learning in Industry
**Work on project
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-3''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-8''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-10 to 11-24=====

*'''11-10''' ''Applying the Guidelines; KLI & Selecting appropriate instructional principles''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***Do post for the reading.
** Time permitting: Peer review of instructional design

*'''11-15''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-17''' ''Finish discussion of experimentation''

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

Educational Research Methods 2017

2016-10-24T14:22:01Z

Koedinger: Created page with "===Research Methods for the Learning Sciences 05-748=== Spring 2014 Syllabus Carnegie Mellon University ====Class times==== 4:30 to 5:50 Tuesday & Thursday ====Location====..."

===Research Methods for the Learning Sciences 05-748===
Spring 2014 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
5312 Wean Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 14 (T)
* Choosing Qualitative & Quantitative Methods: Jan 16 (R)
* Video and Verbal Protocol Analysis: Jan 21, 23, 28, 30, Feb 4,6 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 11, 13, 18, 20 (TRTR)
* Educational Design Research: Feb 25 (R)
* Educational Measurement & Psychometrics: Feb 27, Mar 4, 6 (TRT)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 11, 13 (TR)
* Surveys, Questionnaires, Interviews: Mar 18, 20 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 25, 27, Apr 1 (TRT)
* Flex day: Apr 3 (R)
* Educational Data Mining & Causal Inference: Apr 8, 15, 17 (TTR)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 10 (R)
* Experimental Methods: Apr 22, 24, 29 (TRT)
* Wrap-up: May 1 (R)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-14
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-16 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 21]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 23 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 28 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Jan 30 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 4 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 6 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-11 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-13 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-18 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-20 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-25

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-27

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-4

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-6 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-11 and 3-13 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
* [Plans for these classes were communicated by Kiesler (& Koedinger) via email.]
*3-18
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-20
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-25
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-27
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-1
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-3 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-8
**Before class on 4-8, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-10 NO CLASS - Spring Carnival

*4-15
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-17
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

=====Experimental Research Methods (Koedinger)=====
*4-22 Continuation of Causal Inference
*4-24
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*4-29
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-1
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 9.

Educational Research Methods 2014

2016-10-24T14:21:41Z

Koedinger:

A new version of this course is [[Educational Research Methods 2017]].

===Research Methods for the Learning Sciences 05-748===
Spring 2014 Syllabus Carnegie Mellon University

====Class times====
4:30 to 5:50 Tuesday & Thursday

====Location====
5312 Wean Hall

====Instructor====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

====Class URLs====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014
learnlab.org/research/wiki/index.php/Educational_Research_Methods_2014

For reading reports: [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

===Goals===
The goals of this course are to learn data collection, design, and analysis methodologies that are particularly useful for scientific research in education. The course will be organized in modules addressing particular topics including cognitive task analysis, qualitative methods, protocol and discourse analysis, survey design, psychometrics, educational data mining, and experimental design. We hope students will learn how to apply these methods to their own research programs, how to evaluate the quality of application of these methods, and how to effectively communicate about using these methods.

===Course Prerequisites===
To enroll you must have taken 85-738, "Educational Goals, Instruction, and Assessment" or get the permission of the instruction.

===Textbook and Readings===
"The Research Methods Knowledge Base: 3rd edition" by William M.K. Trochim and James P. Donnelly. You can find it at [http://www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160 www.atomicdogpublishing.com/BookDetails.asp?BookEditionID=160]

The course registration id is 1620032912010.

Other readings will be assigned in class. See below.

===Flipped Homework: Reading Reports and Pre-Class Assignments===

We are often going to implement "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps students (you!) to "problematize" the topic -- to get a better
sense of what you don't know and what questions you have. It helps instructors focus the class discussion to better avoid belaboring what students already know and to better pursue student needs and interests.

Students will be asked to write "reading reports" before most class sessions. We will use the discussion board on Blackboard ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]) for this purpose.

Unless otherwise directed by instructors, students should make '''two posts''' on the readings '''before 9am''' on the day of class that those readings are due. If slides for the class are available, please review these as well.

These posts serve multiple purposes: 1) to improve your understanding and learning from the readings, 2) to provide instructors with insight into what aspects of the readings merit further discussion, either because of student need or interest, and 3) as an incentive to do the readings before class!

In general, please come to class prepared to ask questions and give answers.

Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research

*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

You may be asked to do other activities before class, such as answer questions on-line using the [http://assistment.org Assistment system], parts of the an [http://oli.web.cmu.edu/openlearning/ OLI course], or beginning work on an assignment. That way you can come to class with a better appreciation for what you do not understand and need to learn.

===Grading===

There will be assignments associated with each section of the course. Grades will be determined by your performance on these assignments, by before-class preparation activities including reading reports, by your participation in class, and by a final paper.

* Course work
** 30% Before-class preparation, including reading reports, and in-class participation
** 40% Assignments
* Project & final paper - Initial ideas due Feb 15, research question and likely data source due March 30 [satisfied by posting on Blackboard], Final paper due May 10.
** 30% Design a new study based on one or more of these methods that pushes your own research in a new direction.
:# Apply a method from the class to your research. You should not choose a method that you already know well. Because some methods will be introduced after the project proposal date, we are open to a modification in your project to apply the newly introduced method. But, please check with us to get feedback and approval on a proposed change.
:# No more than 15 double-spaced pages. Be efficient. Space is always limited in academic publications and you will find it useful to learn to include only what is important. You can frame your write-up as though the audience were reviewers of a grant proposal or an internal project proposal. As you would in a grant proposal, please include some literature review and discussion of significance of the area you want to investigate. You should also briefly detail plans for participants, explain specifically how you will apply the method, and describe how you will analyze the data.

===Class Schedule in Brief===
* Formulating Good Research Questions: Jan 14 (T)
* Choosing Qualitative & Quantitative Methods: Jan 16 (R)
* Video and Verbal Protocol Analysis: Jan 21, 23, 28, 30, Feb 4,6 (TRTRTR)
** Guest Instructors: Marsha Lovett & Carolyn Rose
* Performing Cognitive Task Analysis: Feb 11, 13, 18, 20 (TRTR)
* Educational Design Research: Feb 25 (R)
* Educational Measurement & Psychometrics: Feb 27, Mar 4, 6 (TRT)
** Guest Instructor: Brian Junker
* NO CLASS – Spring break, Mar 11, 13 (TR)
* Surveys, Questionnaires, Interviews: Mar 18, 20 (TR)
** Guest Instructor: Sara Kiesler
* Educational Data Mining & Learning Curves: March 25, 27, Apr 1 (TRT)
* Flex day: Apr 3 (R)
* Educational Data Mining & Causal Inference: Apr 8, 15, 17 (TTR)
** Guest Instructor: Richard Scheines
* NO CLASS – Spring Carnival, Apr 10 (R)
* Experimental Methods: Apr 22, 24, 29 (TRT)
* Wrap-up: May 1 (R)

===Class Schedule with Readings and Assignments===
'''NOTE:''' This is a "living" document. It carries over elements from the past course offering that may get changed before the scheduled class period.

=====Course Intro, Research Questions, Picking Methods (Koedinger)=====
*1-14
**Read Trochim Chapter 1, particularly sections 1-2d and 1-4. See above for how to get the book -- [[Media:Trochim-Ch01.pdf|but here's Chapter 1.]]
**Do the chpt 1 quiz
**[[Media:CourseIntroGoodQuestions14.ppt|Lecture slides 2014]]

*1-16 Choosing Qualitative & Quantitative Methods
**Read Trochim Chapter 6 on Qualitative Methods. Please order the book, but one last time [[Media:Trochim-Ch06.pdf|here's Chapter 6 if you need it.]]
**Do the chpt 6 quiz
**Read Koedinger, K.R., Booth, J.L., & Klahr, D. (2013). [[Media:InstructionalComplexity2013.pdf‎|Instructional complexity and the science to constrain it]]. ''Science, 342'', 935-937. [[Media:InstructionalComplexity2013.pdf‎|PDF]]
**[Optional reading] Nathan, M., & Alibali, M. (2010). Learning sciences. WIREs Cognitive Science. [[Media:Nathan&Alibali_2010_WIREs_LS.pdf|PDF]]

** Draft Table relating research purposes and methods: [https://docs.google.com/spreadsheet/ccc?key=0AmjMq6vN8egedFlBVmkwV3A4dWNzeHNsNGlqc00yQVE]

=====Video and Verbal Protocol Analysis (Lovett, Rosé) =====

The 2014 plan for these six sessions is in [[Media:PIERResearchMethodsPlan2014.doc|this document]].

By the end of this module, students should be able to:
*Explain what is involved in collecting and analyzing verbal data (including both “hand” and automatic approaches to analysis)
*Recognize when – and explain why – protocol analysis is/is not appropriate to particular research situations.
*Apply protocol analysis methods to already collected and segmented data.

Besides reading and discussing articles, students will complete a coding scheme design assignment.

Four parts of this assignment will be done as homework or in-class work:
*Part A (homework): Between sessions 2 and 3, propose one or more hypotheses and think about how you could use protocol analysis on the given data set to evaluate those hypotheses.
*Part B (homework): By session 5, develop a short coding manual and apply your coding scheme to a subset of the provided data. Bring 2 printouts to class. Also install LightSIDE software on your laptop and make sure it runs (http://ankara.lti.cs.cmu.edu/side/download.html).
*In class Part C: In session 5, swap coding manuals with a classmate and use their coding manual to code the same data they have coded (but not looking at their codes!), and measure reliability.
*Part D (homework): For session 6, prepare data for automatic coding, and bring soft-copy to class along with your laptop.

*Session 1[Jan 21]: Connecting discussion and learning

*In this session we will explore the connection between discussion and learning, specifically investigating how stylistic aspects of language use enable or constrain articulation of ideas at different levels of abstraction, and how they affect how students position themselves or are positioned within an academic discourse. We will explore these issues in connection with different theoretical perspectives on learning including cognitive, sociocognitive, and sociocultural.

*If this is your first exposure to this material, focus mainly on the Howley et al. chapter. If this is your second exposure, skim the Howley et al chapter and focus mainly on the Adamson et al. article and the comparison between the two.

**Howley, I., Mayfield, E. & Rosé, C. P. (2013). Linguistic Analysis Methods for Studying Small Groups, in Cindy Hmelo-Silver, Angela O’Donnell, Carol Chan, & Clark Chin (Eds.) International Handbook of Collaborative Learning, Taylor and Francis, Inc.[[http://www.learnlab.org/research/wiki/images/5/58/Chapter-Methods-Revised-Final.pdf]]

**Adamson, D., Dyke, G., Jang, H. J., Rosé, C. P. (2014). Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs, International Journal of AI in Education 24(1), pp91-121. [[http://www.learnlab.org/research/wiki/images/e/ea/SpecialIssueAdamson-ThirdRevision_Accepted.pdf]]

*Discussion Questions (pick 2 or 3 of these to discuss as they relate to your reading focus):
**What do you see as the advantages and disadvantages of adopting methods from linguistics for the analysis of verbal data from studies of student learning?
**In the Howley chapter, the role of discussion in learning as it is conceptualized within a variety of theoretical frameworks was compared and contrasted. Which do you agree most with and why?
**Pick one of the conversation extracts from the chapter and critique the provided analysis from the perspective of your chosen theoretical framework.
**How could protocol analysis be used to shed light on what was happening in one or more of the the Adamson et al., 2013 studies?
**What do you see as the trade offs between the style of automated process analysis used in the Adamson et al. article and the more linguistically motivated approach discussed in the Howley et al article?

*Session 2[Jan 23 Carolyn]: Overview of Protocol Analysis

**In this discussion, we will begin to explore the basics of collecting verbal protocol data as well as a high-level view of what’s involved in analyzing such data. Whereas the focus in the initial session was on theory, the focus here will be on methodology of protocol analysis by hand. We will explore different uses of verbal data.

**Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 63), 271-315.
[[http://chilab.asu.edu/papers/Verbaldata.pdf]]

**Discussion Questions:
***What are the main contrasts between the approach Chi advocates for analysis of verbal data and how she presents verbal protocol analysis?
***What can be gained from using these approaches? Which if either do you have experience with, and if so, can you explain that experience?
***How does Chi present these methodologies as complementary to more formally quantitative methodologies?

**Example Coding Manual [[http://www.learnlab.org/research/wiki/images/9/9c/Negotiation_10.pdf]]

*Session 3[Jan 28 Marsha]: Practical aspects of analyzing verbal data

**In this session we will break down the process of designing a coding scheme into practical steps.

**Gihooly, K. J., Fioratou, E., Anthony, S. H., Wynn, V. (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects, British Journal of Psychology, 98, pp 611-625. [[http://www.learnlab.org/research/wiki/images/c/c9/GihoolyEtAl2007.pdf]]

**van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994).The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. New York: Academic Press. Chapter 7 [[http://www.learnlab.org/research/wiki/images/archive/6/63/20130125191704%21VanSch7.pdf]]

*Discussion Questions:
**What, if any, of the steps involved in protocol analysis did you find confusing?
**Which of these steps would you say are most methodologically challenging? most theoretically important?
**How might the steps differ for individual, talk-aloud data vs. collaborative, chat data?

*Session 4[Jan 30 Carolyn]: Methodological considerations related to manual and automatic analysis

**Here we will discuss issues related to reliability and validity, and efficiency of analysis. We will also contrast different types of protocol analyses, namely categorical types of analyses versus word counting approaches.

**Rosé, C. P., Wang, Y.C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F., (2008). Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning, International Journal of Computer Supported Collaborative Learning [[http://www.learnlab.org/research/wiki/images/0/0e/Rose_Analyzing_Collaborative.pdf]]

*Discussion Questions:
**What do you see as the trade-offs between the style of protocol analysis illustrated in this article and that from Adamson et al.?
**What was the most surprising result you read about in the paper? How do the capabilities you read about compare with what you would expect to be able to do with automatic analysis technology?
**What role can you imagine automatic analysis of verbal data playing in your research? Where would it fit within your research process?
**What do you think is the most important caveat related to automatic analysis described in the paper?

*Session 5[Feb 4 Marsha]: Inter-Rater Reliability and When to Use Protocol Data

**In this lecture, we will discuss issues of reliability for protocol data (how to compute Cohen’s kappa and how to resolve coding disagreements). We will also discuss the conditions under which verbal protocol data are/are not appropriate.

**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 1-31). Cambridge, MA: The MIT Press. [Introduction and Summary][[http://www.learnlab.org/research/wiki/images/archive/b/b8/20130125181231%21ProtAna1.pdf]]
**Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis (pp. 78-107). Cambridge, MA: The MIT Press. [Effects of Verbalization] [[http://www.learnlab.org/research/wiki/images/archive/f/fe/20130125181401%21ProtAnalysis2.pdf]]

*Discussion Questions:
**What are the key features that make verbal protocols appropriate/not?
**What can researchers do to collect and analyze such data most effectively?

*Session 6[Feb 6 Carolyn and Marsha]: Tools For Supporting Protocol Analysis

**In this session we will introduce some new technology for facilitating protocol analysis tasks. Students will gain hands on experience with a new technology called SIDE Tools [[http://ankara.lti.cs.cmu.edu/side/download.html]]. You will work with the data you coded in the last session. Please read the user’s manual.

*Discussion Questions:
**What evidence do you as a human use to distinguish between the codes in your coding scheme? How much of this evidence do you think a computer would be able to take advantage of?
**Looking at your coded data, which aspects do you predict will be easy to automatically code, and which do you think will be too hard?

=====Cognitive Task Analysis (CTA) (Koedinger) =====
*2-11 Empirical Cognitive Task Analysis (CTA) via Structured Interviews of Experts
**[[Media:Clark CTA In Healthcare Chapter 2012.pdf |Clark et al (2012) on Cognitive Task Analysis and improving instruction]]
**Do two posts on Blackboard.
***One point of reflection for you on the Clark et al reading is to compare and contrast with recommendations for collection and analysis from van Someren et al and from Ericsson et al. (If you saw Bror Saxberg's PIER talk last year, you may have heard that Kaplan is using CTA, with Clark's advice, to revise and improve their courses.)
**Optional readings:
***[[Media:Lovett01CandI.pdf|Kinds of CTA and instructional design by Marsha Lovett]]
***[[Media:Feldon_Timmerman_etal_2010.pdf|CTA for improving instruction of Biology research by David Feldon]]

*2-13 Rational CTA via Cognitive Modeling
**Zhu X., Lee Y., Simon H.A., & Zhu, D. (1996). Cue recognition and cue elaboration in learning from examples. In Proceedings of the National Academy of Sciences 93, (pp. 1346±1351). [[Media:PNAS-1996-Zhu-Simon.pdf|PNAS-1996-Zhu-Simon.pdf]]
**[Optional reading] Chapter 2: How Experts Differ From Novices in Bransford, J. D., Brown, A., & Cocking, R. (2000). (Eds.), How people learn: Mind, brain, experience and school (expanded edition). Washington, DC: National Academy Press. [[Media:HowPeopleLearnCh2.pdf|HowPeopleLearnCh2.pdf]]
***Besides being an interesting read, a key point of this reading is the nature of expert knowledge (declarative and procedural) and how it is highly "conditionalized". Their discussion of adaptive expertise is also important and interesting.
**[Optional reading] Zhu, X. & Simon, H. A. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4(3), 137-166. [[Media:Zhu&Simon-1987.pdf|Zhu&Simon-1987.pdf]]

*2-18 Doing CTA for higher-level thinking/learning skills
**Azevedo et al on think alouds during learning from hypermedia [[Media:AzevedoMoosJohnson&Chauncey2010.pdf|AzevedoMoosJohnson&Chauncey2010.pdf]]
**Aleven, V., McLaren, B., Roll, I., & Koedinger, K. R. (2004). Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. In J.C. Lester, R.M. Vicari, & F. Parguacu (Eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, 227-239. Berlin: Springer-Verlag. [[Media:AlevenITS2004.pdf|AlevenITS2004.pdf]]
**Klahr, D., & Carver, S.M. (1988). Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. Cognitive Psychology, 20, 362-404. [[Media:Klahr&carver88.pdf|Klahr&carver88.pdf]]
***Pick '''one''' of these readings to focus on and skim the other two. Target your first post on that reading (and make clear which one it was). Your second post can be on any of the three. These readings illustrate the use of Cognitive Task Analysis (CTA) for higher level thinking and learning skills. The Klahr & Carver reading shows how CTA can facilitate the design of instruction that achieves a substantial level of transfer. The Azevedo et al and Aleven et al readings provide examples of CTA at the level of metacognitive skills or learning skills. When you skim all three, pay particular attention to 1) what are tasks the authors are analyzing, 2) what is their goal, 3) what is(are) the method(s) of analysis, and 4) what modeling approaches do the authors use to represent the output of their analysis: Do they use any of production rules, goal trees, semantic nets, hierarchical task models, or other?
*Other possible readings:
**Kinds of CTA and instructional design: Lovett [[Media:Lovett01CandI.pdf|Lovett01CandI.pdf]]
**Relevant to cognitive modeling: Newell & Simon [[Media:Human_Problem_Solving.pdf|Human_Problem_Solving.pdf]]
**A form of CTA with young kids: Siegler, R.S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8 (4), 481-520, Elsevier. [[Media:Siegler76.pdf|Siegler76.pdf]]

*2-20 Empirical quantitative CTA via Difficulty Factors Assessment
**Read: Koedinger, K.R. & Nathan, M.J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. ''The Journal of the Learning Sciences, 13'' (2), 129-164. [[Media:Koedinger-Nathan-LS04.pdf|Koedinger-Nathan-LS04.pdf]]
***In addition to think aloud, another empirical approach to Cognitive Task Analysis is to compare student performance on a space of similar tasks designed to test specific hypotheses about the knowledge demands of those tasks. We have called this approach "Difficulty Factors Assessment" and the Koedinger & Nathan paper is an early example. The former assignment below, which is focused on rational CTA, provides an example of the similarity in the logic of contrast used in Difficulty Factors Assessment and the contrast between the two tasks or solutions one can do in a rational CTA. Skim Koedinger & MacLaren to see another example of a production rule model and of a method of quantitative evaluation of that model by fitting it to coding categories from a solution protocol analysis.
**Skim: Koedinger, K.R., & MacLaren, B. A. (2002). Developing a pedagogical domain theory of early algebra problem solving. CMU-HCII Tech Report 02-100. Accessible via http://reports-archive.adm.cs.cmu.edu/hcii.html [[Media:KoedingerMacLaren02.pdf|KoedingerMacLaren02.pdf]]
**Do two posts on these readings.

**Other optional readings
***[[Media:Applying-CTA-assignment.docx|See prior CTA assignment.]]
***Koedinger, K.R. & McLaughlin, E.A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), ''Proceedings of the 32nd Annual Conference of the Cognitive Science Society.'' (pp. 471-476.) Austin, TX: Cognitive Science Society. [[Media:Koedinger-mclaughlin-cs2010.pdf|Koedinger-mclaughlin-cs2010.pdf]]
***Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The case of fraction division: In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum. [[Media:Rittle-Johnson-Koedinger-cogsci01.pdf|Rittle-Johnson-Koedinger-cogsci01.pdf]]
***Koedinger, K. R., Corbett, A. C., & Perfetti, C. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. ''Cognitive Science''. [[Media:KLI-paper-v5.13.pdf|KLI-paper-v5.13.pdf]]

=====Psychometrics, reliability, Item Response Theory (Junker)=====

*NEW ASSIGNMENTS [Plans for these classes were communicated by Brian Junker via email.]

*2-25

**Quick introduction to the R statistical language

**Please complete and bring comments & questions to class on Tues Feb 28.
**Please download research_methods_r_assignment.zip from http://www.stat.cmu.edu/~brian/PIER-methods/. The Zip file contains three further files:
*** R-preassignment.pdf - instructions for this assignment
*** r-tutorial-1.R - examples of statistical things that you will do in R, for this assignment
*** thermo11_data_integrated.csv - a data set for the examples.

*2-27

1. From Trochim:

A. Chapter 3 - the vocabulary of measurement

B. Chapter 5 - on constructing scales (it's ok to focus
on the material up through sect 5.2a; the rest is
more of a skim [but I'd be happy to talk about that
in class also])

2. On item response theory (IRT), a set of statistical models that are used
to construct scales and to derive scores from them, especially in education
and psychological research:

A. [[Media:Harris-article.pdf|Harris Article (PDF)]]

Please take and self-score the test at the end of
this article. Count each part of question one as
one point, and each of the remaining three questions
as one point (no partial credit!). Bring your 8
scores to class. E.g. if you missed 1(c) and (d), and
you also missed question 4, then you would bring to
class the following scores:

1 1 0 0 1 1 1 0

If you missed 1(a) and (b) and question 2, bring the
following scores:

0 0 1 1 1 0 1 1

(note that the total score is 5 in both cases, but
the pattern of rights and wrongs differs; it is the
pattern that we are interested in).

B. Please browse *online* through pp 1-23 of the pdf at
[http://www.metheval.uni-jena.de/irt/VisualIRT.pdf].

The math is a bit heavy going but there are links
to apps that illustrate various points in the
harris article.

So skim the math and play with the apps.

*3-4

The assignment for this lecture has two parts.

** (A) An R assignment TBA. This you can actually email to my by Fri Mar 7.
** (B) The readings below.

On Tue we will discuss whatever of A and/or B seem interesting

1. "Psychometric Principles in Student Assessment" by Mislevy et al ([[Media:mislevy-principles-2001.pdf|Mislevy (PDF)]])

Read through p 18. This is a more modern modern look at some of
the same issues that are addressed in Trochim's chapters.

The remainder of this paper surveys various probabilistic models
for the "measurement model" portion of Mislevy's framework (Figure
1). It is quite interesting but we will not pursue it.

2. "Cognitive Assessment Models with Few Assumptions..." by Junker & Sijtsma ([[Media:junker-sijtsma-apm-2001.pdf|Junker, Sijtsma (PDF)]])

Please read up through p 266 only.

The math is a bit heavy going so please try to read around it to
see what the point of the article is.

We will try to look at some of the data in the article as examples
in lecture 2.

*3-6 Continued discussion of Psychometrics [moved Design Research as option for Flex Day]

=====NO CLASS – Spring break 3-11 and 3-13 =====

=====Surveys, Questionnaires, Interviews (Kiesler) =====
* [Plans for these classes were communicated by Kiesler (& Koedinger) via email.]
*3-18
**Reading: Trochim Ch 4 and 5
***You already read Ch 5 for the Psychometric section, so just review it. For both chapters, answer Trochim's on-line questions before and/or after reading (answering the questions before gives you goals for reading). For discussion board posts, do one post on how have or might use a survey (e.g., of student attitudes) in your own research. Make another post about Chapter 4, such as something you learned, a question you have, or an answer to someone else's question.
*3-20
**Do the following homework assignment [[Media:Arm-modQuestEduc.doc]]. Sara directs: Keep the text that's there and fill in answers, working through it step by step. I'm just as interested in your revisions as in the final version. Est time 45 minutes.
**Readings
***Tourangeau, Roger, and T. Yan. 2007. "Sensitive questions in surveys." Psychological Bulletin, 133(5): 859-883. [[Media:Tourangeau_SensitiveQuestions.pdf]]
***Tourangeau, R. (2000). “Remembering what happened: Memory errors and survey reports.@ In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for research and practice (pp. 29-48). Englewood Cliffs, N.J.: Lawrence Erlbaum. [[Media:Tourangeau_RememberingWhatHappened.pdf]]

=====Educational Data Mining -- Learning Curve Analysis (Koedinger) =====
*3-25
**BRING YOUR LAPTOP FOR ALL THESE SESSIONS
**Two in-class activities: 1) Make progress toward your course project (e.g., further write-up of your research question, justify method selection, search for relevant data) and 2) Work on learning curve assignment (due on Thursday by 9am).
***Start on the assignment BEFORE CLASS and complete up to step B4, requesting access to the data.
**Read the following paper and make two posts as usual.
***Stamper, J. & Koedinger, K.R. (2011). Human-machine student model discovery and improvement using data. In J. Kay, S. Bull & G. Biswas (Eds.), Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353-360. Berlin: Springer.[[Media:Stamper-Koedinger-AIED2011.pdf| Stamper-Koedinger-AIED2011.pdf]]
***'''Optional:'''Ritter, F.E., & Schooler, L. J. (2001). The learning curve. In W. Kintch, N. Smelser, P. Baltes, (Eds.), International Encyclopedia of the Social and Behavioral Sciences. Oxford, UK: Pergamon. [[Media:RittterSchooler01.pdf | RittterSchooler01.pdf]]
**'''Assignment:''' The assignment ([[Media:Learning-curve-assignment-2014.doc | Learning-curve-assignment-2014.doc]]) is a tutorial on using DataShop to begin analyzing learning curves. Upload to Blackboard (or email to me) by 9am on Thursday.
*3-27
**Read the following paper and make two posts as usual.
***Koedinger, K.R., McLaughlin, E.A., & Stamper, J.C. (2012). Automated student model improvement. In Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., & Stamper, J. (Eds.), Proceedings of the 5th International Conference on Educational Data Mining, pp. 17-24. [[Media:KoedingerMcLaughlinStamperEDM12.pdf|KoedingerMcLaughlinStamperEDM12.pdf]]
**In-class activity: Start on one of the two exercises (A or B) below. Provide a brief writeup in response to each of the numbered steps and include a summary of the result you achieved (e.g., did you get a more predictive model as measured by AIC, BIC, or cross validation). Turn in this writeup and the supporting file (KC model table or R file) on Blackboard. Make significant progress before class next Tuesday (get to a point where you are stuck or can see your way to the end). Due by end of day on Wednesday, 4-2.
*4-1
**In-class: Bring your laptop to work on (finish!) your chosen exercise (A or B).
**Read the following paper and make two posts as usual.
***Zhang, X., Mostow, J., & Beck, J. E. (2007, July 9). All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. AIED2007 Educational Data Mining Workshop, Marina del Rey, CA [[Media:AIED2007_EDM_Zhang_ld_transfer.pdf|AIED2007_EDM_Zhang_ld_transfer.pdf]]
***'''Optional:''' Roberts, Seth, & Pashler, Harold. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358 - 367. [[Media:2000_roberts_pashler.pdf]]
***'''Optional:''' Schunn, C. D., & Wallach, D. (2005). Evaluating goodness-of-fit in comparison of models to data. In W. Tack (Ed.), Psychologie der Kognition: Reden and Vorträge anlässlich der Emeritierung von Werner Tack (pp. 115-154). Saarbrueken, Germany: University of Saarland Press. [[Media:GOF.doc]]

Do A or B:
A. Modify a KC model in a DataShop dataset
1. What is the DataShop dataset you modified?
2. Describe how you used the HMST procedure (from Stamper paper)
to identify a KC to try to improve
3. Show how you recoded that KC with new KCs (turn in your modified
KC file) & describe why you made the change you did
4. After importing your new KC model to DataShop, did it improve the
predictions (are any of the metrics, AIC, BIC, or cross validation)?
(Caution: Make sure your new KC model labels the same number of
observations as the KC model you are modifying.)

B. Use R to create an alternative statistical model to AFM
1. Approximate afm in R using either glm or lmer. How do the parameter
estimates and metrics (AIC and BIC) compare with results in DataShop?
2. Modify the regression equation to try to improve the prediction.
Some options include: a) adding a student by KC interaction (there
are just main effects of student and KC in AFM), b) adding student
slopes (there is just a KC slope in AFM), c) counting success and
failure opportunities separately (both kinds of opportunities are
lumped together in AFM), d) using log of Opportunity, e) including
step (perhaps as a random effect) ...
3. Turn in your R file including metrics (log-liklihood, parameters,
AIC, BIC) on the statistical models you compared
4. Summarize whether or not your modification changes model fit (log
liklihood), changes the number of parameters (from what to what),
and, most importantly, improves prediction (as measured by AIC or BIC)

===== Flex day (Koedinger) =====
*4-3 To be used in case of rescheduling, for a student-driven topic, and/or for Review of Projects or Past Topics
** We will wrap up on EDM for learning curves (option1) and, time permitting, give work time for your project.
***Option1. More on Educational Data Mining
***Option2. Return to Design Research & Qualitative Methods (Koedinger)
***Trochim Ch 8 (stop before 8.5), Ch 13 (stop before 13.3)
***Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1). [[Media:2004 Barab Squire.pdf|PDF]]
***Optional reading: Chapter on Design Research in Handbook of Learning Sciences

=====Educational Data Mining -- Causal Inference from Data (Scheines) =====
*4-8
**Before class on 4-8, do Unit 2 in the OLI course Empirical Research Methods
Go to: http://oli.cmu.edu/learn-with-oli/see-our-free-open-courses/
Scroll down and click on the rightmost tab, "Prior work (5)"
Click on "Empirical Research Methods" and then on "[Enter Course]"
Click on "CMU users sign in here" to login with your CMU account
or "Enter Without an Account"
Complete "UNIT 2: Regression, Prediction and Causation"
**See this website for relevant material: http://www.hss.cmu.edu/philosophy/casestudiesworkshop.php (It is for a workshop on "Case Studies of Causal Discovery with Model Search")
**Scroll down to the schedule. Videos and slides are posted for most of the talks. Three that are relevant to this class are:
***a) Tutorial on causal learning (my tutorial on Tetrad)
***b) Educational Research I (overview of causal discovery in educational research)
***c) Educational Research II (Martina explaining the paper you are assigned)
**There are also case studies from economics, fMRI, genetics, biology, as well as educational research.

*4-10 NO CLASS - Spring Carnival

*4-15
**Read and post about Scheines, R., Leinhardt, G., Smith, J., and Cho, K. (2005). Replacing lecture with web-based course materials. Journal of Educational Computing Research, 32, 1, 1-26. [[Media:Scheines jecr revised.doc | PDF]]

*4-17
**Read and post about [[Media:RauScheinesAlevenRummel_EDM2013_camera-ready_final.pdf | Rau, Scheines, Aleven, & Rummel (2013]]

=====Experimental Research Methods (Koedinger)=====
*4-22 Continuation of Causal Inference
*4-24
**Reading: Trochim Ch 7 and 9
**Do two posts on Blackboard.
**OLD Slides: [[Media:L02_--_Basic_Research_and_Experimental_Methods.ppt|Experimental_Methods.ppt]] and [[Media:L03-True-Experiments.ppt|True-Experiments.ppt]]
*4-29
**Reading: Trochim Ch 10
**OLD Slides: [[Media:L04-quasi-experiments.ppt|Quasi-Experiments.ppt]]
*5-1
**Reading: Trochim Ch 14
**Optional: Try ANOVA module of OLI Statistics course

=====Wrap-up=====
If needed, schedule a course wrap-up

Final project is due May 9.

E-Learning Design Principles and Methods 2016

2016-10-20T12:58:13Z

Koedinger: /* Multimedia Principles 10-27 to 11-8 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?
*** Do posts for the readings.

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''Midterm review''
**Do review quiz and bring questions to class
**No new post or quiz.
**Optional Readings:
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''

=====Multimedia Principles 10-27 to 11-8=====
*'''10-27''' ''[Optional topic: Work on project, CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-3''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-8''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-10 to 11-24=====

*'''11-10''' ''Applying the Guidelines; KLI & Selecting appropriate instructional principles''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***Do post for the reading.
** Time permitting: Peer review of instructional design

*'''11-15''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-17''' ''Finish discussion of experimentation''

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2016-10-13T12:47:29Z

Koedinger: /* Multimedia Principles 10-27 to 11-24 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?
*** Do posts for the readings.

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''Midterm review''
**Do review quiz and bring questions to class
**No new post or quiz.
**Optional Readings:
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''

=====Multimedia Principles 10-27 to 11-8=====
*'''10-27''' ''[Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-3''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-8''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-10 to 11-24=====

*'''11-10''' ''Applying the Guidelines; KLI & Selecting appropriate instructional principles''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***Do post for the reading.
** Time permitting: Peer review of instructional design

*'''11-17''' ''Finish discussion of experimentation''

*'''11-15''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-17''' ''Finish discussion of experimentation''

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2016-10-13T12:47:06Z

Koedinger: /* Putting it together & evaluation 11-15 to 11-24 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?
*** Do posts for the readings.

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''Midterm review''
**Do review quiz and bring questions to class
**No new post or quiz.
**Optional Readings:
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''

=====Multimedia Principles 10-27 to 11-24=====
*'''10-27''' ''[Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-3''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-8''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-10 to 11-24=====

*'''11-10''' ''Applying the Guidelines; KLI & Selecting appropriate instructional principles''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***Do post for the reading.
** Time permitting: Peer review of instructional design

*'''11-17''' ''Finish discussion of experimentation''

*'''11-15''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-17''' ''Finish discussion of experimentation''

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2016-10-13T12:43:23Z

Koedinger: /* Multimedia Principles 10-27 to 11-24 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?
*** Do posts for the readings.

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''Midterm review''
**Do review quiz and bring questions to class
**No new post or quiz.
**Optional Readings:
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''

=====Multimedia Principles 10-27 to 11-24=====
*'''10-27''' ''[Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-3''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-8''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-15 to 11-24=====

*'''11-15''' ''Applying the Guidelines''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
***Do post for the reading.
** Time permitting: KLI Review; Peer review of instructional design

*'''11-17''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2016-10-13T12:40:22Z

Koedinger: /* Learning By Doing Principles 10-4 to 10-25 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?
*** Do posts for the readings.

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''Midterm review''
**Do review quiz and bring questions to class
**No new post or quiz.
**Optional Readings:
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''

=====Multimedia Principles 10-27 to 11-24=====
*'''10-27''' ''[Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1'''

*'''11-3''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-8''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-10''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-15 to 11-24=====

*'''11-15''' ''Applying the Guidelines''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
***Do post for the reading.
** Time permitting: KLI Review; Peer review of instructional design

*'''11-17''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2016-09-29T13:00:52Z

Koedinger: /* Multimedia Principles 10-27 to 11-24 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''KLI & Selecting appropriate instructional principles; Midterm review''
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***No post or quiz for this reading.

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''
**Optional Readings:
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]

=====Multimedia Principles 10-27 to 11-24=====
*'''10-27''' ''[Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1'''

*'''11-3''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-8''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-10''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

=====Putting it together & evaluation 11-15 to 11-24=====

*'''11-15''' ''Applying the Guidelines''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
***Do post for the reading.
** Time permitting: KLI Review; Peer review of instructional design

*'''11-17''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2016-09-27T12:52:11Z

Koedinger: /* Cognitive Task Analysis (CTA) 9-20 to 9-29 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
***Do Discussion Board post on reading.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Peer review of P2
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''KLI & Selecting appropriate instructional principles; Midterm review''
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***No post or quiz for this reading.

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''
**Optional Readings:
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]

=====Multimedia Principles 10-27 to 11-24=====
*'''10-27''' ''[Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1'''

*'''11-3''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-8''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-10''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-15''' ''Applying the Guidelines; KLI Review; Peer review of instructional design '''[too much on one day?]'''''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
***Do post for the reading.

*'''11-17''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2016-09-01T12:37:59Z

Koedinger: /* Instructional Goals and Assessment 9-6 to 9-15 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: Review of e-learning examples
***BRING a print-out of your e-learning examples to class
***We will find examples of promises & pitfalls, knowledge component types, etc.

*'''9-8''' ''Writing assessments to meet goals; Evidence-centered design''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
**Reading: [[Media:Evidence-centered-design-2003.pdf|Evidence-centered design]]
***Do TWO discussion board posts on readings.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Work on P3. Analyzing your data
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''KLI & Selecting appropriate instructional principles; Midterm review''
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***No post or quiz for this reading.

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''
**Optional Readings:
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]

=====Multimedia Principles 10-27 to 11-24=====
*'''10-27''' ''[Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1'''

*'''11-3''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-8''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-10''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-15''' ''Applying the Guidelines; KLI Review; Peer review of instructional design '''[too much on one day?]'''''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
***Do post for the reading.

*'''11-17''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

E-Learning Design Principles and Methods 2016

2016-09-01T12:36:23Z

Koedinger: /* E-Learning Introduction 8-30 to 9-1 */

====Course Details====
Course number: 05-823

Semester: Fall 2016

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 4301

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 4th edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
For the syllabus go to [http://www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no/ www.learnlab.org/research/wiki/index.php?title=E-Learning_Design_Principles_and_Methods_2016&redirect=no]

For quizzes and reading reports go to [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard]

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 8am'' on the day of class. Quizzes can be taken as many times as you want and your score will be the last attempt before 8am.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy (updated 8/30/16) ====

Given that class discussion is a major part of the course, you should only use laptops, cell phones, and smart phones when you are directed to do so or within the common note taking Google page that I will provide. Deviation form this policy will result in a reduction in your participation grade. You will often need or want a laptop or smart device. You will definitely need one during testing days (marked as such on the schedule).

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2016.docx|Project assignment]] '''(Submit project steps preferably as a Google document (provide access by sharing with the instructor and TA), but a Word document is OK. Do not submit a pdf.)'''
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 8-30 to 9-1
**Aug 30 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 1 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-6 to 9-15
**Sept 6 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Sept 8 Writing assessments to meet goals; Evidence-centered design
**Sept 13 Why data toward goal setting improves design
**Sept 15 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-20 to 9-29
**Sept 20 Empirical CTA: Structured Interviews
**Sept 22 Think Alouds & Rational CTA
**Sept 27 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 Quantitative CTA via Data Mining
*Learning By Doing Principles 10-4 to 10-25
**Oct 4 3.Evidence-based practice; KLI Learning & Instructional Events
**Oct 6
**Oct 11 12.Does Practice Make Perfect; 14.Who’s in Control?
**Oct 13 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Oct 18 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Oct 20 KLI & Selecting appropriate instructional principles; Midterm review
**Oct 25 Midterm exam '''Bring laptop to class'''
*Multimedia Principles 10-7 to 11-24
**Oct 27 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]
**Nov 1
**Nov 3 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Nov 8 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Nov 10 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Nov 15 17.Applying the Guidelines; KLI Review; Peer review of instructional design [too much on one day?]
**Nov 17 In vivo experimentation; A/B Testing
**Nov 22 Flex topic; Presentation & Report Preparation
**Nov 24 Thanksgiving, no class
*Final & Project Presentations 11-29 to 12-8
**Nov 29 Project Presentations
**Dec 1 Project Presentations
**Dec 6 Project Presentations
**Dec 8 Final Exam '''Bring laptop to class'''
*Final Project due Dec 12
*If needed: Final Exam Make-up -TBD

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 8-30 to 9-1=====

*'''8-30''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:Ch1-4th_edition.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard Try to do this quiz before Tues class but it must be completed before Thurs class.
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment2016.docx|Examples assignment]] is due next Mon, Sept 5. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2016.docx|Project]] step 1 is due in 16 days on Thursday, 9-15
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-1''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:Ch2-4th_edition.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 9-1"
**Class activity: Project idea discussion
***Be ready to discuss some of your ''preliminary project ideas'' -- enter these in Google Doc of class notes
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-6 to 9-15=====

*'''9-6''' ''Determining instructional goals; KLI KCs; Bloom's taxonomy''
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: KC type in e-learning examples

*'''9-8''' ''Writing assessments to meet goals; Evidence-centered design''
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
**Reading: [[Media:Evidence-centered-design-2003.pdf|Evidence-centered design]]
***Do TWO discussion board posts on readings.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*'''9-13''' ''Why data toward goal setting improves design''
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*'''9-15''' ''Online assessment; Practice e-assessment implementation''
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit all project steps as a shared google doc or in Blackboard as a Word doc. Do not submit a pdf.)'''
**Assignment: P2 is due Sept 29

=====Cognitive Task Analysis (CTA) 9-20 to 9-29=====
*'''9-20''' ''Empirical CTA: Structured Interviews''
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*'''9-22''' ''Think Alouds & Rational CTA''
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''9-27''' ''Quantitative Cognitive Task Analysis: Difficulty Factors Assessment''
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*'''9-29''' ''Quantitative CTA via Data Mining; CTA to improve instructional design''
**Class activity: Work on P3. Analyzing your data
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 13

=====Learning By Doing Principles 10-4 to 10-25 =====
*'''10-4''' ''Evidence-based practice; KLI Learning & Instructional Events''
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sections 4-5 (12 pages)
*** Do quizzes for the readings.

*'''10-6''' ''Practice, practice, practice''
**Reading: [[Media:Make_It_Stick_Ch1-2.pdf|Make It Stick Ch 1 and Ch 2 (45 pages) ]]
**Recommended reading: [[Media:Visible_Learning_Ch_9.pdf|Visible Learning Ch 9 (38 pages) ]]
** Additional Reading: [[Media:Scheines_2005.pdf|Scheines paper (20 pages) ]]
**Class activity: Work on P3. How will you collect data?

*'''10-11''' ''Does Practice Make Perfect; Who’s in Control?''
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*'''10-13''' ''E-Learning to Build Problem Solving Skill; Simulations and Games''
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 31

*'''10-18''' ''Segmenting and Pretraining; Leveraging Examples in E-Learning''
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*'''10-20''' ''KLI & Selecting appropriate instructional principles; Midterm review''
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***No post or quiz for this reading.

*'''10-25''' ''Midterm exam'' '''Bring laptop to class'''
**Optional Readings:
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]

=====Multimedia Principles 10-27 to 11-24=====
*'''10-27''' ''[Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]''
**DUE Mon, 10/31 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-17

*'''11-1'''

*'''11-3''' ''Multimedia Principle; Contiguity Principle; Practice applying''
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*'''11-8''' ''Modality Principle & Redundancy Principle; Practice applying''
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-10''' ''Coherence Principle & Personalization Principle; Practice applying''
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.

*'''11-15''' ''Applying the Guidelines; KLI Review; Peer review of instructional design '''[too much on one day?]'''''
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
***Do post for the reading.

*'''11-17''' ''In vivo experimentation; A/B Testing''
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 11-29

*'''11-22''' ''Flex topic; Presentation & Report Preparation''

*'''11-24''' ''Thanksgiving, no class''

=====Final & Project Presentations 11-29 to 12-18=====

*'''11-29''' ''Project Presentations''
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-12. It should include the reflection statement (see the project assignment handout).
*'''12-1''' ''Project Presentations''
*'''12-6''' ''Project Presentations''
*'''12-8''' Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - TBD

====='''Final Project Due 12-12'''=====

Main Page

2016-07-23T01:00:08Z

Koedinger: /* Robust Learning Theoretical Framework */

== Robust Learning Theoretical Framework ==

A key goal of LearnLab is to support learning scientists in providing explanations of results using, as much as possible, the same core terminology and addressing an accumulating body of precise theoretical [[Instructional Principles and Hypotheses|principles of instruction]]. While a single theory of learning may emerge in the long term, the immediate goal is to encourage researchers to maximize the overlap between each others' theories. We want to help the field get beyond the "[[Toothbrush Problem]]" in theorizing.

In 2012, we published a theoretical framework, called the [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Knowledge-Learning-Instruction (KLI) framework]. This framework builds on our 2006 [http://learnlab.org/clusters/PSLC_Theory_Frame_June_15_2006.pdf theoretical framework document] and, importantly, on the contributions to this wiki. In 2013, we published [http://pact.cs.cmu.edu/pubs/Koedinger-Science-2013.pdf a paper in Science] reviewing instructional principles, the complexity of their combinations, and recommendations for LearnLab style research to address this complexity.

In order to display the integration across research projects, this wiki maintains multiple theoretical hierarchies, one of [[Instructional Principles and Hypotheses|Instructional Principles]] and another of empirical studies, which are found on the cluster pages: [[Coordinative Learning]], [[Interactive Communication]], and [[Refinement and Fluency]].

Many other on-line lists of instructional and learning principles exist ['''''please add more!''''']:
* [http://ies.ed.gov/ncer/pubs/practiceguides/20072004.asp Organizing Instruction and Study to Improve Student Learning], one of the Practice Guides of the Department of Education, Institute for Education Sciences. See also:
** [http://dww.ed.gov/topic/topic_landing.cfm?PA_ID=9&T_ID=19&Tab=1 The web page "Doing What Works: Psychology of Learning"]
** [http://www.nctq.org/dmsStage/Learning_About_Learning_Report A web-based update of this guide]
* [http://www.psyc.memphis.edu/learning/principles/ Principles of Learning] from Lifelong Learning at Work and at Home, a subgroup of the Association for Psychological Science.
* [http://www.edu-design-principles.org Design Principles Database] maintained by the NSF-funded [http://www.telscenter.org/ TELS (Technology Enhanced Learning in Science)] project
* [http://www.cmu.edu/teaching/principles/ Principles of Teaching and Learning] from CMU's Eberly Center for Teaching and Learning
* [http://en.wikipedia.org/wiki/Principles_of_learning Wikipedia entry for principles of learning]
* [http://www.udlcenter.org/aboutudl/udlguidelines Universal Design for Learning guidelines]
* See a review of principles and the role of technology in refining them in [https://www.youtube.com/watch?v=ysWTEWx9L_A this talk by Art Graesser].

[http://pact.cs.cmu.edu/koedinger/Koedinger10_WMV%20V9.wmv Ken Koedinger HCII Seminar Talk: Why Designing Effective Learning Interactions Is Not Easy and How We Can Do Better: Part I]

=== Toward a Hierarchical Structure of Robust Learning Hypotheses and Findings ===

The structure of the research cluster and study pages is as follows (a template can be found at [[Project Page Template and Creation Instructions]]). When a set of explanations share many terms and hypotheses, we make a node for each explanation, make a node for their common features, and link the nodes so that the common-feature node is the parent of each explanation node. In most cases a "node" is a single wiki page, but sometimes a node involves several wiki pages.

In order to more clearly display the integration, each node contains:
#An ''abstract'' that briefly describes the research encompassed by the node;
#A ''glossary'' that defines terms used elsewhere in this node but not defined in the nodes that are parents, grandparents, etc. of this node;
#The ''research question'' stated as concisely as possible, usually in a single sentence;
#A ''background and significance'' section that briefly summarizes prior work on the research question and why it is important to answer it;
#The ''dependent variables'', which are observable and typically measure competence, motivation, interaction, meta-learning, or some other pedagogically desirable outcome;
#The ''independent variables'', which typically include instructional environment, activity or method (the instructional "treatment" vs. "control"), and perhaps some student individual difference variables, such as gender or first language;
#The ''hypothesis'', which is a concise statement of the relationship among the variables that answers the research question;
#The ''findings'', which are the results of the study if it has been performed or the expected findings from the study if it has not -- explicitly indicate if the findings are preliminary;
#An ''explanation'' that describes the theoretical rationale for the hypothesis using the PSLC theoretical framework. It should be a paragraph or two and provide a causal chain that mentions mediating variables -- unobservable, hypothetical attributes of the students (e.g., knowledge components or path choices), how the treatment affects these, and how they, in turn, affect the dependent variables;
#The ''descendants'', which lists links to descendant nodes of this one, if there are any;
#A ''further information'' section that points to documents using hyper links and/or references in APA format. Each indicates briefly the document's relationship to the node (e.g., whether the document is a paper reporting the node in full detail, a proposal describing the motivation and design of the study in more detail, the node for a similar PSLC research study, etc.).

Experience suggests that the glossaries carry much of the load in explaining the research, and that carefully defining and exemplifying terms often pays off later in reducing confusion and facilitating collaboration. Consequently, the glossaries are sometimes so long that they are spit off as separate wiki pages.

The [[Root_node|root node of the hierarchy]] represents the overarching research question of how to achieve robust learning particularly in academic settings, like K12 schools and colleges. The question is necessarily abstract and is not the sort of question that can actually be tested by a single decisive experiment.

The immediate descendants of the root node are three nodes representing somewhat more specific research questions. There are nodes for each of [[Coordinative Learning]], [[Interactive Communication]] and [[Refinement and Fluency]]. These present somewhat more concrete research questions. They are specializations to the overarching questions, and form a bridge to testable hypotheses posed by individual research projects.

The leaves of the hierarchy (i.e., nodes with no descendants) represent individual research studies. A leaf node can also represent a group of studies or a whole project if the activities are sufficiently similar that it makes sense to summarize them with a single node. Each leaf node is maintained by its project’s leader and may or may not be publicly accessible depending on the state of the research.

Between the cluster nodes and the leaves, there may be some intervening nodes. For instance, if a group of Coordinative Learning studies all address a similar research question (e.g., how to use verbal and visual instruction together effectively), then a node may be created to summarize their shared aspects. Its parent is the Coordinative Learning cluster node, and its descendants are the relevant project nodes.

For a draft of [[Macro-level framework|an alternative framework based on independent dimensions of instruction click here]].

Knowledge component

2016-05-31T19:34:10Z

Koedinger: Added links to KLI Framework and DataShop

=== Knowledge Component ===

A knowledge component is a description of a mental structure or process that a learner uses, alone or in combination with other knowledge components, to accomplish [[Step|steps]] in a task or a problem. A full description and taxonomy of knowledge components can be found in [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Koedinger, Corbett, & Perfetti (2012) ]. A knowledge component is a generalization of everyday terms like concept, principle, fact, or skill, and cognitive science terms like [[schema]], production rule, misconception, or facet. When we say a student "has" a knowledge component, it might mean the student can describe it in words (e.g., "Vertical angles are congruent") or it might simply mean that the student behaves as described by the knowledge component, but may not be able to describe it themselves. In this second case, to say the student "has" the knowledge component "If angle A and B are vertical angles and angle A is X degrees, then angle B is X degrees" means the student will behave in accord with it even though they might not be able to state the rule. The first is an "explicit" knowledge component, like a fact or principle, and the second an "implicit" knowledge component , like a skill. Much of what first language learners know about their first language involves implicit knowledge components.

A knowledge component (KC) relates [[Features|features]] to a response where both the features and response(s) can be either external, in the world, like cues in a stimulus and a motor response or internal, in the mind, like inferred features and a new goal.

KCs are "correct" when all of the features are relevant to making the response and none of them are irrelevant. In geometry, for example, the knowledge component "if angles look equal, then conclude they are equal" is incorrect because it includes an irrelevant feature "angles look equal" and is missing a relevant feature like "the angles are at the base of an isosceles triangle". See also [[Feature validity|feature validity]] and [[Refinement|refinement]].

An example of a knowledge component analysis (a kind of [[cognitive task analysis]]) can be found in the description of Julie Booth's study [[Booth|knowledge component construction vs. recall]]. In her case, the key knowledge components are concepts and skills for making decisions during problem solving in the domain of algebra equation solving. She identifies both incorrect knowledge components that students tend to acquire and correct knowledge components that good students eventually acquire.

A data mining approach to knowledge component analysis is support by [http://learnlab.org/DataShop LearnLab's DataShop].

=== Kinds of knowledge components ===
Mental representations of:
* Domain knowledge
** Facts, concepts, principles, rules, procedures, strategies
* Prerequisite knowledge
** Feature encoding knowledge (see examples in [[Booth|Algebra]] and [[Applying optimal scheduling of practice in the Chinese Learnlab|Chinese radicals]])
* Integrative knowledge
** Schemas or procedures that connect other KCs
* Metacognitive knowledge
** About knowledge, controlling use or acquisition of knowledge (see the [[The Help Tutor Roll Aleven McLaren|help-seeking project]])
*Beliefs & interests
** What one likes, believes

=== Cross-cutting distinctions ===
* Correct vs. incorrect
* Verbal (explicit) vs. non-verbal (implicit)
* Probabilistic vs. discrete

=== Not knowledge components ===
* Any external representation of knowledge
** Like textbook descriptions or an example
* Generic cognitive structures
** Working memory
* Continuous parameters on knowledge representations
** Strength, level of engagement, implicit value of a goal, affect

=== Other uses of "knowledge" ===
“Knowledge” in PSLC is used as in the Cognitive Science and AI traditions. The mind is a knowledge base stored in the brain’s hardware. All competencies and behaviors are determined by “knowledge” in this sense. "Knowledge" in philosophy is “justified true belief" whereas our use of knowledge components includes both incorrect (false) knowledge and implicit (no explicit belief or justification) knowledge. "Knowledge" in education is basic facts (1st level of Bloom’s (1956) taxonomy) whereas knowledge components can be procedures, integrating schemas, complex reasoning strategies, metacognitive skills …, that is, all levels of Bloom’s taxonomy.

=== References ===
* VanLehn, K. (2006). The behavior of tutoring systems. ''International Journal of Artificial Intelligence in Education'', 16 (3), 227-265 [http://www.pitt.edu/~vanlehn/Stringent/Abstracts/06IJAIED.htm Abstract&PDF]

* Koedinger's PSLC Lunch Talk from August, 2006.

* Norma Chang's CMU Psychology PhD Thesis (2006) on surface vs. structural problem variations and resultant acquisition of relevant vs. irrelevant features ("spurious correlations" with surface features).

* Bloom, B. S. (Ed.). (1956). Taxonomy of educational objectives. Handbook 1: Cognitive domain. New York: McKay.

[[Category:Glossary]] [[Category:PSLC_General]] [[Category:DataShop_Glossary]]

Knowledge component

2016-05-31T19:32:26Z

Koedinger: Added links to KLI Framework and DataShop

=== Knowledge Component ===

A knowledge component is a description of a mental structure or process that a learner uses, alone or in combination with other knowledge components, to accomplish [[Step|steps]] in a task or a problem. A full description and taxonomy of knowledge components can be found in [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Koedinger, Corbett, & Perfetti (2012) ]. A knowledge component is a generalization of everyday terms like concept, principle, fact, or skill, and cognitive science terms like [[schema]], production rule, misconception, or facet. When we say a student "has" a knowledge component, it might mean the student can describe it in words (e.g., "Vertical angles are congruent") or it might simply mean that the student behaves as described by the knowledge component, but may not be able to describe it themselves. In this second case, to say the student "has" the knowledge component "If angle A and B are vertical angles and angle A is X degrees, then angle B is X degrees" means the student will behave in accord with it even though they might not be able to state the rule. The first is an "explicit" knowledge component, like a fact or principle, and the second an "implicit" knowledge component , like a skill. Much of what first language learners know about their first language involves implicit knowledge components.

A knowledge component (KC) relates [[Features|features]] to a response where both the features and response(s) can be either external, in the world, like cues in a stimulus and a motor response or internal, in the mind, like inferred features and a new goal.

KCs are "correct" when all of the features are relevant to making the response and none of them are irrelevant. In geometry, for example, the knowledge component "if angles look equal, then conclude they are equal" is incorrect because it includes an irrelevant feature "angles look equal" and is missing a relevant feature like "the angles are at the base of an isosceles triangle". See also [[Feature validity|feature validity]] and [[Refinement|refinement]].

An example of a knowledge component analysis (a kind of [[cognitive task analysis]]) can be found in the description of Julie Booth's study [[Booth|knowledge component construction vs. recall]]. In her case, the key knowledge components are concepts and skills for making decisions during problem solving in the domain of algebra equation solving. She identifies both incorrect knowledge components that students tend to acquire and correct knowledge components that good students eventually acquire.

A data mining approach to knowledge component analysis is support by [http://learnlag.org/DataShop LearnLab's DataShop].

=== Kinds of knowledge components ===
Mental representations of:
* Domain knowledge
** Facts, concepts, principles, rules, procedures, strategies
* Prerequisite knowledge
** Feature encoding knowledge (see examples in [[Booth|Algebra]] and [[Applying optimal scheduling of practice in the Chinese Learnlab|Chinese radicals]])
* Integrative knowledge
** Schemas or procedures that connect other KCs
* Metacognitive knowledge
** About knowledge, controlling use or acquisition of knowledge (see the [[The Help Tutor Roll Aleven McLaren|help-seeking project]])
*Beliefs & interests
** What one likes, believes

=== Cross-cutting distinctions ===
* Correct vs. incorrect
* Verbal (explicit) vs. non-verbal (implicit)
* Probabilistic vs. discrete

=== Not knowledge components ===
* Any external representation of knowledge
** Like textbook descriptions or an example
* Generic cognitive structures
** Working memory
* Continuous parameters on knowledge representations
** Strength, level of engagement, implicit value of a goal, affect

=== Other uses of "knowledge" ===
“Knowledge” in PSLC is used as in the Cognitive Science and AI traditions. The mind is a knowledge base stored in the brain’s hardware. All competencies and behaviors are determined by “knowledge” in this sense. "Knowledge" in philosophy is “justified true belief" whereas our use of knowledge components includes both incorrect (false) knowledge and implicit (no explicit belief or justification) knowledge. "Knowledge" in education is basic facts (1st level of Bloom’s (1956) taxonomy) whereas knowledge components can be procedures, integrating schemas, complex reasoning strategies, metacognitive skills …, that is, all levels of Bloom’s taxonomy.

=== References ===
* VanLehn, K. (2006). The behavior of tutoring systems. ''International Journal of Artificial Intelligence in Education'', 16 (3), 227-265 [http://www.pitt.edu/~vanlehn/Stringent/Abstracts/06IJAIED.htm Abstract&PDF]

* Koedinger's PSLC Lunch Talk from August, 2006.

* Norma Chang's CMU Psychology PhD Thesis (2006) on surface vs. structural problem variations and resultant acquisition of relevant vs. irrelevant features ("spurious correlations" with surface features).

* Bloom, B. S. (Ed.). (1956). Taxonomy of educational objectives. Handbook 1: Cognitive domain. New York: McKay.

[[Category:Glossary]] [[Category:PSLC_General]] [[Category:DataShop_Glossary]]

Main Page

2016-05-12T16:58:15Z

Koedinger:

== Robust Learning Theoretical Framework ==

A key goal of LearnLab is to support learning scientists in providing explanations of results using, as much as possible, the same core terminology and addressing an accumulating body of precise theoretical [[Instructional Principles and Hypotheses|principles of instruction]]. While a single theory of learning may emerge in the long term, the immediate goal is to encourage researchers to maximize the overlap between each others' theories. We want to help the field get beyond the "[[Toothbrush Problem]]" in theorizing.

In 2012, we published a theoretical framework, called the [http://pact.cs.cmu.edu/pubs/KLI-KoedingerCorbettPerfetti2012-pre.pdf Knowledge-Learning-Instruction (KLI) framework]. This framework builds on our 2006 [http://learnlab.org/clusters/PSLC_Theory_Frame_June_15_2006.pdf theoretical framework document] and, importantly, on the contributions to this wiki. In 2013, we published [http://pact.cs.cmu.edu/pubs/Koedinger-Science-2013.pdf a paper in Science] reviewing instructional principles, the complexity of their combinations, and recommendations for LearnLab style research to address this complexity.

In order to display the integration across research projects, this wiki maintains multiple theoretical hierarchies, one of [[Instructional Principles and Hypotheses|Instructional Principles]] and another of empirical studies, which are found on the cluster pages: [[Coordinative Learning]], [[Interactive Communication]], and [[Refinement and Fluency]].

Many other on-line lists of instructional and learning principles exist ['''''please add more!''''']:
* [http://ies.ed.gov/ncer/pubs/practiceguides/20072004.asp Organizing Instruction and Study to Improve Student Learning], one of the Practice Guides of the Department of Education, Institute for Education Sciences. See also [http://dww.ed.gov/topic/topic_landing.cfm?PA_ID=9&T_ID=19&Tab=1 the web page "Doing What Works: Psychology of Learning"].
* [http://www.psyc.memphis.edu/learning/principles/ Principles of Learning] from Lifelong Learning at Work and at Home, a subgroup of the Association for Psychological Science.
* [http://www.edu-design-principles.org Design Principles Database] maintained by the NSF-funded [http://www.telscenter.org/ TELS (Technology Enhanced Learning in Science)] project
* [http://www.cmu.edu/teaching/principles/ Principles of Teaching and Learning] from CMU's Eberly Center for Teaching and Learning
* [http://en.wikipedia.org/wiki/Principles_of_learning Wikipedia entry for principles of learning]
* [http://www.udlcenter.org/aboutudl/udlguidelines Universal Design for Learning guidelines]
* See a review of principles and the role of technology in refining them in [https://www.youtube.com/watch?v=ysWTEWx9L_A this talk by Art Graesser].

[http://pact.cs.cmu.edu/koedinger/Koedinger10_WMV%20V9.wmv Ken Koedinger HCII Seminar Talk: Why Designing Effective Learning Interactions Is Not Easy and How We Can Do Better: Part I]

=== Toward a Hierarchical Structure of Robust Learning Hypotheses and Findings ===

The structure of the research cluster and study pages is as follows (a template can be found at [[Project Page Template and Creation Instructions]]). When a set of explanations share many terms and hypotheses, we make a node for each explanation, make a node for their common features, and link the nodes so that the common-feature node is the parent of each explanation node. In most cases a "node" is a single wiki page, but sometimes a node involves several wiki pages.

In order to more clearly display the integration, each node contains:
#An ''abstract'' that briefly describes the research encompassed by the node;
#A ''glossary'' that defines terms used elsewhere in this node but not defined in the nodes that are parents, grandparents, etc. of this node;
#The ''research question'' stated as concisely as possible, usually in a single sentence;
#A ''background and significance'' section that briefly summarizes prior work on the research question and why it is important to answer it;
#The ''dependent variables'', which are observable and typically measure competence, motivation, interaction, meta-learning, or some other pedagogically desirable outcome;
#The ''independent variables'', which typically include instructional environment, activity or method (the instructional "treatment" vs. "control"), and perhaps some student individual difference variables, such as gender or first language;
#The ''hypothesis'', which is a concise statement of the relationship among the variables that answers the research question;
#The ''findings'', which are the results of the study if it has been performed or the expected findings from the study if it has not -- explicitly indicate if the findings are preliminary;
#An ''explanation'' that describes the theoretical rationale for the hypothesis using the PSLC theoretical framework. It should be a paragraph or two and provide a causal chain that mentions mediating variables -- unobservable, hypothetical attributes of the students (e.g., knowledge components or path choices), how the treatment affects these, and how they, in turn, affect the dependent variables;
#The ''descendants'', which lists links to descendant nodes of this one, if there are any;
#A ''further information'' section that points to documents using hyper links and/or references in APA format. Each indicates briefly the document's relationship to the node (e.g., whether the document is a paper reporting the node in full detail, a proposal describing the motivation and design of the study in more detail, the node for a similar PSLC research study, etc.).

Experience suggests that the glossaries carry much of the load in explaining the research, and that carefully defining and exemplifying terms often pays off later in reducing confusion and facilitating collaboration. Consequently, the glossaries are sometimes so long that they are spit off as separate wiki pages.

The [[Root_node|root node of the hierarchy]] represents the overarching research question of how to achieve robust learning particularly in academic settings, like K12 schools and colleges. The question is necessarily abstract and is not the sort of question that can actually be tested by a single decisive experiment.

The immediate descendants of the root node are three nodes representing somewhat more specific research questions. There are nodes for each of [[Coordinative Learning]], [[Interactive Communication]] and [[Refinement and Fluency]]. These present somewhat more concrete research questions. They are specializations to the overarching questions, and form a bridge to testable hypotheses posed by individual research projects.

The leaves of the hierarchy (i.e., nodes with no descendants) represent individual research studies. A leaf node can also represent a group of studies or a whole project if the activities are sufficiently similar that it makes sense to summarize them with a single node. Each leaf node is maintained by its project’s leader and may or may not be publicly accessible depending on the state of the research.

Between the cluster nodes and the leaves, there may be some intervening nodes. For instance, if a group of Coordinative Learning studies all address a similar research question (e.g., how to use verbal and visual instruction together effectively), then a node may be created to summarize their shared aspects. Its parent is the Coordinative Learning cluster node, and its descendants are the relevant project nodes.

For a draft of [[Macro-level framework|an alternative framework based on independent dimensions of instruction click here]].

E-Learning Design Principles and Methods 2016

2015-11-18T17:31:25Z

Koedinger: /* Final & Project Presentations 12-1 to 12-10 */

====Course Details====
Course number: 05-823

Semester: Fall 2015

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 5222

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 3rd edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/E-learning_Design_Principles_2014 learnlab.org/research/wiki/index.php/E-learning_Design_Principles_2014]

For quizzes and reading reports go [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard].

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 9am'' on the day of class.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy====

Given that class discussion is a major part of the course, laptops, cell phones, and smart phones are not to be used in class during Lecture days. Failure to listen to this will result in a reduction in your participation grade. During testing days (marked as such on the schedule), however, you will need your laptop.

Students have the option of using a laptop during presentations '''only if''' they are doing so to take notes and submit those notes to the full class for example on blackboard. To facilitate note taking during Lecture days, lecture slide handouts may be provided, if requested.

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2015.docx|Project assignment]] (Submit project steps preferable as a Google document, but a Word document is OK.)
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 9-1- to 9-3
**Sept 1 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 3 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-8 to 9-17
**Sept 8 Determining instructional goals; Bloom's taxonomy; KLI KCs; Practice on goal setting
**Sept 10 Standards & Assessment Frameworks; Evidence-based design; Practice assessment writing
**Sept 15 Goal-setting Interviews: Structured Interviews, Contextual Inquiry; Practice interviewing
**Sept 17 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-22 to 10-1
**Sept 22 Rational & Empirical CTA via Think Alouds
**Sept 24 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 CTA to improve model building & instructional design
**Oct 1 Quantitative CTA via Data Mining
*Multimedia Principles 10-6 to 10-22
**Oct 6 3.Evidence-based practice; KLI Learning & Instructional Events;
**Oct 8 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Oct 13 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Oct 15 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Oct 20 Flex topic; Midterm review
**Oct 22 Midterm exam '''Bring laptop to class'''
*Learning By Doing Principles 10-27 to 11-24
**Oct 27 KLI & Selecting appropriate instructional principles
**Oct 29 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Nov 3 12.Does Practice Make Perfect; 14.Who’s in Control?
**Nov 5 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Nov 10 17.Applying the Guidelines; KLI Review
**Nov 12 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?] '''(need substitute for this class)'''
**Nov 17 Peer review of instructional design
**Nov 19 In vivo experimentation; A/B Testing
**Nov 24 Flex topic; Presentation & Report Preparation
**Nov 26 Thanksgiving, no class
*Final & Project Presentations 12-1 to 12-10
**Dec 1 Final Exam '''Bring laptop to class'''
**Dec 3 Project Presentations
**Dec 8 Project Presentations
**Dec 10 Project Presentations
*Final Project due Dec 14
*If needed: Final Exam Make-up - Thurs Dec 17, 1-4pm in GHC 5222

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 9-1 to 9-3=====

*'''9-1''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:E-Learning-Ch1.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment.docx|Examples assignment]] is due next Mon, Sept 7. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2015.docx|Project]] step 1 is due in 16 days on Thursday, 9-17
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-3''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:E-Learning-Ch2.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 10-3"
***Slides for this chapter are [[Media:L02-how-people-learn+instr-complexity.pptx|here]].
**Class activity: Promises & pitfalls review of e-learning examples
***BRING a print-out of an e-learning example to class
**Class activity: Project idea discussion
***Come prepared with a ''preliminary project idea'' -- post one or more project ideas in Discussion Board forum "Project idea feedback & partner solicitation"
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-8 to 9-17=====

*9-8 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: KC type in e-learning examples

*9-10 Writing assessments to meet goals; Evidence-centered design
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
**Reading: [[Media:Evidence-centered-design-2003.pdf|Evidence-centered design]]
***Do TWO discussion board posts on readings.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*9-15 Why data toward goal setting improves design
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*9-17 Online assessment; Practice e-assessment implementation
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit as a Word document.)'''
**Assignment: P2 is due Oct 1

=====Cognitive Task Analysis (CTA) 9-22 to 10-1=====
*9-22 Empirical CTA: Structured Interviews
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*9-24 Think Alouds & Rational CTA
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*9-29 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*10-1 Quantitative CTA via Data Mining; CTA to improve instructional design
**Class activity: Work on P3. Analyzing your data
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 15

=====Multimedia Principles 10-6 to 10-22=====
*10-6 Evidence-based practice; KLI Learning & Instructional Events
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sectionts 4-5 (12 pages)
*** Do quizzes for the readings.

*10-8 Multimedia Principle; Contiguity Principle; Practice applying
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*10-13 Modality Principle & Redundancy Principle; Practice applying
**Class activity: Work on P3. How will you collect data?
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*10-15 Coherence Principle & Personalization Principle; Practice applying
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.
**Due: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 29

*10-20 Flex topic; Midterm Review

*10-22 Midterm exam '''Bring laptop to class'''
**Optional Readings:
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]

=====Learning By Doing Principles 10-27 to 11-24 =====

*10-27 KLI & Selecting appropriate instructional principles
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***No post or quiz for this reading.

*10-29 Segmenting and Pretraining; Leveraging Examples in E-Learning
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE Mon, 11/2 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-19

*11-3 Does Practice Make Perfect; Who’s in Control?
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*11-5 E-Learning to Build Problem Solving Skill; Simulations and Games
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*11-10 Applying the Guidelines; KLI Review
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
***Do post for the reading.

*11-12 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]

*11-17 Peer review of instructional design

*11-19 In vivo experimentation; A/B Testing
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 12-1

*11-24Flex topic; Presentation & Report Preparation

*11-26 Thanksgiving, no class

=====Final & Project Presentations 12-1 to 12-10=====

*12-1 Project Presentations
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-14. It should include the reflection statement (see the project assignment handout).
*12-3 Project Presentations
*12-8 Project Presentations
*12-10 Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - Thurs Dec 17, 1-4pm in GHC 5222

=====Final Project Due on 12-14=====

E-Learning Design Principles and Methods 2016

2015-11-18T16:42:46Z

Koedinger: /* Grading */

====Course Details====
Course number: 05-823

Semester: Fall 2015

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 5222

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 3rd edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/E-learning_Design_Principles_2014 learnlab.org/research/wiki/index.php/E-learning_Design_Principles_2014]

For quizzes and reading reports go [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard].

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 9am'' on the day of class.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy====

Given that class discussion is a major part of the course, laptops, cell phones, and smart phones are not to be used in class during Lecture days. Failure to listen to this will result in a reduction in your participation grade. During testing days (marked as such on the schedule), however, you will need your laptop.

Students have the option of using a laptop during presentations '''only if''' they are doing so to take notes and submit those notes to the full class for example on blackboard. To facilitate note taking during Lecture days, lecture slide handouts may be provided, if requested.

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2015.docx|Project assignment]] (Submit project steps preferable as a Google document, but a Word document is OK.)
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 9-1- to 9-3
**Sept 1 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 3 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-8 to 9-17
**Sept 8 Determining instructional goals; Bloom's taxonomy; KLI KCs; Practice on goal setting
**Sept 10 Standards & Assessment Frameworks; Evidence-based design; Practice assessment writing
**Sept 15 Goal-setting Interviews: Structured Interviews, Contextual Inquiry; Practice interviewing
**Sept 17 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-22 to 10-1
**Sept 22 Rational & Empirical CTA via Think Alouds
**Sept 24 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 CTA to improve model building & instructional design
**Oct 1 Quantitative CTA via Data Mining
*Multimedia Principles 10-6 to 10-22
**Oct 6 3.Evidence-based practice; KLI Learning & Instructional Events;
**Oct 8 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Oct 13 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Oct 15 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Oct 20 Flex topic; Midterm review
**Oct 22 Midterm exam '''Bring laptop to class'''
*Learning By Doing Principles 10-27 to 11-24
**Oct 27 KLI & Selecting appropriate instructional principles
**Oct 29 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Nov 3 12.Does Practice Make Perfect; 14.Who’s in Control?
**Nov 5 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Nov 10 17.Applying the Guidelines; KLI Review
**Nov 12 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?] '''(need substitute for this class)'''
**Nov 17 Peer review of instructional design
**Nov 19 In vivo experimentation; A/B Testing
**Nov 24 Flex topic; Presentation & Report Preparation
**Nov 26 Thanksgiving, no class
*Final & Project Presentations 12-1 to 12-10
**Dec 1 Final Exam '''Bring laptop to class'''
**Dec 3 Project Presentations
**Dec 8 Project Presentations
**Dec 10 Project Presentations
*Final Project due Dec 14
*If needed: Final Exam Make-up - Thurs Dec 17, 1-4pm in GHC 5222

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 9-1 to 9-3=====

*'''9-1''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:E-Learning-Ch1.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment.docx|Examples assignment]] is due next Mon, Sept 7. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2015.docx|Project]] step 1 is due in 16 days on Thursday, 9-17
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-3''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:E-Learning-Ch2.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 10-3"
***Slides for this chapter are [[Media:L02-how-people-learn+instr-complexity.pptx|here]].
**Class activity: Promises & pitfalls review of e-learning examples
***BRING a print-out of an e-learning example to class
**Class activity: Project idea discussion
***Come prepared with a ''preliminary project idea'' -- post one or more project ideas in Discussion Board forum "Project idea feedback & partner solicitation"
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-8 to 9-17=====

*9-8 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: KC type in e-learning examples

*9-10 Writing assessments to meet goals; Evidence-centered design
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
**Reading: [[Media:Evidence-centered-design-2003.pdf|Evidence-centered design]]
***Do TWO discussion board posts on readings.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*9-15 Why data toward goal setting improves design
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*9-17 Online assessment; Practice e-assessment implementation
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit as a Word document.)'''
**Assignment: P2 is due Oct 1

=====Cognitive Task Analysis (CTA) 9-22 to 10-1=====
*9-22 Empirical CTA: Structured Interviews
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*9-24 Think Alouds & Rational CTA
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*9-29 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*10-1 Quantitative CTA via Data Mining; CTA to improve instructional design
**Class activity: Work on P3. Analyzing your data
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do quiz for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 15

=====Multimedia Principles 10-6 to 10-22=====
*10-6 Evidence-based practice; KLI Learning & Instructional Events
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages) [[Media:L03-ELDP-evidence-KLI-2014.pptx|Slides]]
**Reading: KLI paper sectionts 4-5 (12 pages)
*** Do quizzes for the readings.

*10-8 Multimedia Principle; Contiguity Principle; Practice applying
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages) [[Media:Contiguity2014.pptx|Slides]]
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]
**Optional readings & slides about the Contiguity principle:
***Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity [[Media:Moreno Mayer Modality-Contiguity.pdf|Paper]] [[Media:MorenoMayer1999-1.pptx|Slides]]
***Why Some Material is Difficult to Learn [[Media:Sweller Chandler Why Some Material is Difficult to Learn.pdf |Paper]] [[Media:SwellerChandler1994.pptx|Slides]]

*10-13 Modality Principle & Redundancy Principle; Practice applying
**Class activity: Work on P3. How will you collect data?
**Reading: 6.Modality Principle (18 pages) [[Media:Modality-2014-1.pptx |Slides]]
**Reading: 7.Redundancy Principle (18 pages) [[Media:L12-ELDP-Redundancy&eData-2014.pptx|Slides]]
***Do quizzes for the readings.

*10-15 Coherence Principle & Personalization Principle; Practice applying
**Reading: 8.Coherence Principle (28 pages) [[Media:L12-ELDP-Coherence-2014.pptx|Slides]]
**Reading: 9.Personalization Principle (26 pages) [[Media:L14-ELDP-Personalization-2014.pptx|Slides]]
***Do quizzes for the readings.
**Due: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 29

*10-20 Flex topic; Midterm Review

*10-22 Midterm exam '''Bring laptop to class'''
**Optional Readings:
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]

=====Learning By Doing Principles 10-27 to 11-24 =====

*10-27 KLI & Selecting appropriate instructional principles
**Reading: KLI sections 6-7 [[Media:L17-ELDP-KLI-selecting-principles-for syllabus.pptx|Slides]]
***No post or quiz for this reading.

*10-29 Segmenting and Pretraining; Leveraging Examples in E-Learning
**Reading: 10.Segmenting and Pretraining (18 pages) [[Media:L16-ELDP-Segmenting-Pretraining-2014-for syllabus.pptx|Slides]]
**Reading: 11.Leveraging Examples in E-Learning (28 pages) [[Media:L17-ELDP-Worked-Examples-2014-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.
**DUE Mon, 11/2 by 12 noon: P4: Initial Instructional Design
**Assignment: P5 is due 11-19

*11-3 Does Practice Make Perfect; Who’s in Control?
**Reading: 12.Does Practice Make Perfect (28 pages) [[Media:L18-ELDP-Practice-2014-for_syllabus.pptx|Slides]]
**Reading: 14.Who’s in Control? (30 pages) [[Media:Principles_e-learning_14.pdf |Slides]]
***Do quizzes for the readings.

*11-5 E-Learning to Build Problem Solving Skill; Simulations and Games
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages) [[Media:L22-ELDP-ThinkingSkills-2014-for_syllabus.pptx|Slides]]
**Reading: 16.Simulations and Games (32 pages) [[Media:L23-ELDP-GamesSimulations-2014_for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*11-10 Applying the Guidelines; KLI Review
**Reading: 17.Applying the Guidelines (24 pages) [[Media:L25-ELDPCh17-Guidelines-KLIreview-2014.pptx|Slides]]
***Do post for the reading.

*11-12 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]

*11-17 Peer review of instructional design

*11-19 In vivo experimentation; A/B Testing
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 12-1

*11-24Flex topic; Presentation & Report Preparation

*11-26 Thanksgiving, no class

=====Final & Project Presentations 12-1 to 12-10=====

*12-1 Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - Thurs Dec 17, 1-4pm in GHC 5222
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-14. It should include the reflection statement (see the project assignment handout).

*12-3 Project Presentations
*12-8 Project Presentations
*12-10 Project Presentations (we need to reschedule this, perhaps to 12-11)

=====Final Project Due on 12-14=====

E-Learning Design Principles and Methods 2016

2015-10-01T12:13:57Z

Koedinger: /* Multimedia Principles 10-6 to 10-22 */

====Course Details====
Course number: 05-823

Semester: Fall 2015

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 5222

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 3rd edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/E-learning_Design_Principles_2014 learnlab.org/research/wiki/index.php/E-learning_Design_Principles_2014]

For quizzes and reading reports go [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard].

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 9am'' on the day of class.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy====

Given that class discussion is a major part of the course, laptops, cell phones, and smart phones are not to be used in class during Lecture days. Failure to listen to this will result in a reduction in your participation grade. During testing days (marked as such on the schedule), however, you will need your laptop.

Students have the option of using a laptop during presentations '''only if''' they are doing so to take notes and submit those notes to the full class for example on blackboard. To facilitate note taking during Lecture days, lecture slide handouts may be provided, if requested.

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2015.docx|Project assignment]] (Submit project steps as a Word document.)
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 9-1- to 9-3
**Sept 1 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 3 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-8 to 9-17
**Sept 8 Determining instructional goals; Bloom's taxonomy; KLI KCs; Practice on goal setting
**Sept 10 Standards & Assessment Frameworks; Evidence-based design; Practice assessment writing
**Sept 15 Goal-setting Interviews: Structured Interviews, Contextual Inquiry; Practice interviewing
**Sept 17 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-22 to 10-1
**Sept 22 Rational & Empirical CTA via Think Alouds
**Sept 24 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 CTA to improve model building & instructional design
**Oct 1 Quantitative CTA via Data Mining
*Multimedia Principles 10-6 to 10-22
**Oct 6 3.Evidence-based practice; KLI Learning & Instructional Events;
**Oct 8 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Oct 13 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Oct 15 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Oct 20 Flex topic; Midterm review
**Oct 22 Midterm exam '''Bring laptop to class'''
*Learning By Doing Principles 10-27 to 11-24
**Oct 27 KLI & Selecting appropriate instructional principles
**Oct 29 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Nov 3 12.Does Practice Make Perfect; 14.Who’s in Control?
**Nov 5 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Nov 10 17.Applying the Guidelines; KLI Review
**Nov 12 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?] '''(need substitute for this class)'''
**Nov 17 Peer review of instructional design
**Nov 19 In vivo experimentation; A/B Testing
**Nov 24 Flex topic; Presentation & Report Preparation
**Nov 26 Thanksgiving, no class
*Final & Project Presentations 12-1 to 12-10
**Dec 1 Final Exam '''Bring laptop to class'''
**Dec 3 Project Presentations
**Dec 8 Project Presentations
**Dec 10 Project Presentations
*Final Project due Dec 14
*If needed: Final Exam Make-up - Thurs Dec 17, 1-4pm in GHC 5222

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 9-1 to 9-3=====

*'''9-1''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:E-Learning-Ch1.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment.docx|Examples assignment]] is due next Mon, Sept 7. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2015.docx|Project]] step 1 is due in 16 days on Thursday, 9-17
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-3''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:E-Learning-Ch2.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 10-3"
***Slides for this chapter are [[Media:L02-how-people-learn+instr-complexity.pptx|here]].
**Class activity: Promises & pitfalls review of e-learning examples
***BRING a print-out of an e-learning example to class
**Class activity: Project idea discussion
***Come prepared with a ''preliminary project idea'' -- post one or more project ideas in Discussion Board forum "Project idea feedback & partner solicitation"
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-8 to 9-17=====

*9-8 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: KC type in e-learning examples

*9-10 Writing assessments to meet goals; Evidence-centered design
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
**Reading: [[Media:Evidence-centered-design-2003.pdf|Evidence-centered design]]
***Do TWO discussion board posts on readings.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*9-15 Why data toward goal setting improves design
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*9-17 Online assessment; Practice e-assessment implementation
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit as a Word document.)'''
**Assignment: P2 is due Oct 1

=====Cognitive Task Analysis (CTA) 9-22 to 10-1=====
*9-22 Empirical CTA: Structured Interviews
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*9-24 Think Alouds & Rational CTA
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*9-29 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*10-1 Quantitative CTA via Data Mining; CTA to improve instructional design
**Class activity: Work on P3. Analyzing your data
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do discussion board post(s) for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 15

=====Multimedia Principles 10-6 to 10-22=====
*10-6 Evidence-based practice; KLI Learning & Instructional Events
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages)
**Reading: KLI paper sectionts 4-5 (12 pages)
*** Do quizzes for the readings.
**Class activity: Principles present in e-learning examples

*10-8 Multimedia Principle; Contiguity Principle; Practice applying
**Reading: 4.Multimedia Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages)
*** Do quizzes for the readings.
**Optional readings & slides about the multimedia principle:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]

*10-13 Modality Principle & Redundancy Principle; Practice applying
**Class activity: Work on P3. How will you collect data?
**Reading: 6.Modality Principle (18 pages)
**Reading: 7.Redundancy Principle (18 pages)
***Do quizzes for the readings.

*10-15 Coherence Principle & Personalization Principle; Practice applying
**Reading: 8.Coherence Principle (28 pages)
**Reading: 9.Personalization Principle (26 pages)
***Do quizzes for the readings.
**Due: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 29

*10-20 Flex topic; Midterm Review

*10-22 Midterm exam '''Bring laptop to class'''
**Optional Readings:
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]

=====Learning By Doing Principles 10-27 to 11-24 =====

*10-27 KLI & Selecting appropriate instructional principles
**Reading: KLI sections 6-7
***Do quizzes for the reading.
**Assignment: P4 is due 10-29

*10-29 Segmenting and Pretraining; Leveraging Examples in E-Learning
**Reading: 10.Segmenting and Pretraining (18 pages)
**Reading: 11.Leveraging Examples in E-Learning (28 pages)
***Do quizzes for the readings.
**DUE: P4: Initial Instructional Design
**Assignment: P5 is due 11-19

*11-3 Does Practice Make Perfect; Who’s in Control?
**Reading: 12.Does Practice Make Perfect (28 pages)
**Reading: 14.Who’s in Control? (30 pages)
***Do quizzes for the readings.

*11-5 E-Learning to Build Problem Solving Skill; Simulations and Games
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages)
**Reading: 16.Simulations and Games (32 pages)
***Do quizzes for the readings.

*11-10 Applying the Guidelines; KLI Review
**Reading: 17.Applying the Guidelines (24 pages)
***Do quizzes for the reading.

*11-12 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]

*11-17 Peer review of instructional design

*11-19 In vivo experimentation; A/B Testing
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 12-1

*11-24Flex topic; Presentation & Report Preparation

*11-26 Thanksgiving, no class

=====Final & Project Presentations 12-1 to 12-10=====

*12-1 Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - Thurs Dec 17, 1-4pm in GHC 5222
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-14. It should include the reflection statement (see the project assignment handout).

*12-3 Project Presentations
*12-8 Project Presentations
*12-10 Project Presentations (we need to reschedule this, perhaps to 12-11)

=====Final Project Due on 12-14=====

E-Learning Design Principles and Methods 2016

2015-10-01T10:55:26Z

Koedinger: /* Multimedia Principles 10-6 to 10-22 */

====Course Details====
Course number: 05-823

Semester: Fall 2015

Carnegie Mellon University

=====Class times=====
9:00 to 10:20 Tuesday & Thursday

=====Location=====
Gates Hillman Center (GHC) Room 5222

=====Instructor=====
Professor Ken Koedinger

Office: 3601 Newell-Simon Hall, Phone: 412-268-7667

Email: Koedinger@cmu.edu, Office hours by appointment

Teaching assistant: Mimi McLaughlin Email: mimim@cs.cmu.edu

Admininstrative assistant: Jo Bodnar Email: jobodnar@cs.cmu.edu

=====Course Prerequisites=====
To enroll you must either be in the Masters of Educational Technology and Applied Learning Science (METALS) or get the permission of the instructor.

=====Textbook and Readings=====
"E-Learning and the Science of Instruction: 3rd edition" by Ruth Colvin Clark and Richard E. Mayer.

Other readings will be assigned in class. See below.

=====Class URLs=====
Syllabus and useful links: [http://learnlab.org/research/wiki/index.php/E-learning_Design_Principles_2014 learnlab.org/research/wiki/index.php/E-learning_Design_Principles_2014]

For quizzes and reading reports go [http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard].

====Goals====
This course is about e-learning design principles, the evidence and theory behind them, and how to apply these principles to develop effective educational technologies. It is organized around the book "e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning" by Clark & Mayer with further readings drawn from cognitive science, educational psychology, and human-computer interaction. You will learn design principles 1) for combining words, audio, and graphics in multimedia instruction, 2) for combining examples, explanations, practice and feedback in online support for learning by doing, and 3) for balancing learner versus system control and supporting student metacognition. You will read about the experiments that support these design principles, see examples of how to design such experiments, and practice applying the principles in your own educational technology design project.

====Flipped Homework: Reading Quizzes and Reading Reports ====

You will have "flipped homework", a variation on the flipped classroom idea you might have heard of. Flipped homework is an assignment before a relevant class meeting rather than after it. It helps you to check your understanding of what you read, to practice to enhance your memory (we will talk about the "testing effect" in class), and to get a better sense of what you don't know so you are prepared to ask questions in class. It also helps instructors focus the class discussion to better avoid belaboring known points and pursue student needs and interests.

Before some class sessions, you will asked to do a quiz associated with the assigned book chapter. The quizzes will be on the Blackboard site ([http://www.cmu.edu/blackboard/ www.cmu.edu/blackboard], the course is listed as "Special Topics in HCI"). Before other class sessions, you will be asked to write "reading reports". We will use the discussion board on Blackboard. You should complete assigned quizzes or reading reports ''before 9am'' on the day of class.

For reading reports, the discussion forum post will usually direct you as to how to reply.
If not otherwise directed, you should make '''two posts''' on the readings. Your ''two'' posts may be original or in response to another post (one of both is nice).
*Original posts should contain one or more of the following:
**something you learned from the reading or slides
**a question you have about the reading or slides or about the topic in general
**a connection with something you learned or did previously in this or another course, or in other professional work or research
*Replies should be an on-topic, relevant response, clarification, or further comment on another student’s post.

In general, please come to class prepared to ask questions and give answers.

====Laptop Policy====

Given that class discussion is a major part of the course, laptops, cell phones, and smart phones are not to be used in class during Lecture days. Failure to listen to this will result in a reduction in your participation grade. During testing days (marked as such on the schedule), however, you will need your laptop.

Students have the option of using a laptop during presentations '''only if''' they are doing so to take notes and submit those notes to the full class for example on blackboard. To facilitate note taking during Lecture days, lecture slide handouts may be provided, if requested.

If interested in what educational research says about laptop use in class, or multi-tasking more generally, you might look at (available on the course BlackBoard):

*Fried, C. B. (2008). In-class laptop use and its effects on student learning. Computers & Education, 50, 906–914.

*Kirschner, P. A., & Merriënboer, J. J. V. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48(3), 169–183. doi:10.1080/00461520.2013.80439

*Kraushaar, J. M., & Novak, D. C. (2010). Examining the affects [sic] of student multitasking with laptops during the lecture. Journal of Information Systems Education, 21(2), 241-251.

*Wood, E., Zivcakova, L., Gentile, P., Archer, K., De Pasquale, D., & Nosko, A. (2012). Examining the impact of off-task multi-tasking with technology on real-time classroom learning. Computers & Education, 58(1), 365-374. doi:10.1016/j.compedu.2011.08.02

====Grading ====

* 35% Final Project [[Media:E-learning-project-assignment-2015.docx|Project assignment]] (Submit project steps as a Word document.)
**Six parts of final project
**Final project submission
* 5% E-Learning examples assignment
* 15% Midterm exam
* 15% Pre-class quizzes & reading reports
* 15% Final Exam
* 15% Class participation, including reading summary presentations

====Class Schedule in Brief====
*E-Learning Introduction 9-1- to 9-3
**Sept 1 Overview; Examples Assignment; Project; 1.E-learning (The "1." indicates this is a chapter in the Clark & Mayer book)
**Sept 3 2.How People Learn; Instructional complexities; Project topic brainstorming
*Instructional Goals and Assessment 9-8 to 9-17
**Sept 8 Determining instructional goals; Bloom's taxonomy; KLI KCs; Practice on goal setting
**Sept 10 Standards & Assessment Frameworks; Evidence-based design; Practice assessment writing
**Sept 15 Goal-setting Interviews: Structured Interviews, Contextual Inquiry; Practice interviewing
**Sept 17 Online assessment; Practice e-assessment implementation
*Cognitive Task Analysis (CTA) 9-22 to 10-1
**Sept 22 Rational & Empirical CTA via Think Alouds
**Sept 24 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Sept 29 CTA to improve model building & instructional design
**Oct 1 Quantitative CTA via Data Mining
*Multimedia Principles 10-6 to 10-22
**Oct 6 3.Evidence-based practice; KLI Learning & Instructional Events;
**Oct 8 4.Multimedia Principle; 5.Contiguity Principle; Practice applying
**Oct 13 6.Modality Principle & 7.Redundancy Principle; Practice applying
**Oct 15 8.Coherence Principle & 9.Personalization Principle; Practice applying
**Oct 20 Flex topic; Midterm review
**Oct 22 Midterm exam '''Bring laptop to class'''
*Learning By Doing Principles 10-27 to 11-24
**Oct 27 KLI & Selecting appropriate instructional principles
**Oct 29 10.Segmenting and Pretraining; 11.Leveraging Examples in E-Learning
**Nov 3 12.Does Practice Make Perfect; 14.Who’s in Control?
**Nov 5 15.E-Learning to Build Problem Solving Skill; 16.Simulations and Games
**Nov 10 17.Applying the Guidelines; KLI Review
**Nov 12 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?] '''(need substitute for this class)'''
**Nov 17 Peer review of instructional design
**Nov 19 In vivo experimentation; A/B Testing
**Nov 24 Flex topic; Presentation & Report Preparation
**Nov 26 Thanksgiving, no class
*Final & Project Presentations 12-1 to 12-10
**Dec 1 Final Exam '''Bring laptop to class'''
**Dec 3 Project Presentations
**Dec 8 Project Presentations
**Dec 10 Project Presentations
*Final Project due Dec 14
*If needed: Final Exam Make-up - Thurs Dec 17, 1-4pm in GHC 5222

====Class Schedule with Readings and Assignments====

'''NOTE:''' This section is "living" -- parts will evolve as I get a better sense of your needs.

=====E-Learning Introduction 9-1 to 9-3=====

*'''9-1''' ''Course Objectives & Course Project; The boom in e-learning!''
**Reading (from course book): 1.e-Learning: Promise & Pitfalls (20 pages). [[Media:E-Learning-Ch1.pdf|This chapter is here (click to get)]] but order the book right now!
***Pre-class quiz: Answer questions for Chpt1 Quiz on Blackboard
***Slides for this chapter are [[Media:Chpt1-e-learning-promises-pitfalls-2015.pptx|here]].
**Class activity: Introduce your background and interests in e-learning
**Assignment: [[Media:edtech-example-review-assignment.docx|Examples assignment]] is due next Mon, Sept 7. Please submit on blackboard.
**Assignment: [[Media:E-learning-project-assignment-2015.docx|Project]] step 1 is due in 16 days on Thursday, 9-17
**For next time:
***BRING two screen shots of an e-learning example to ''next'' class
***Review project step 1 and come with a preliminary project idea.
***a) Do the readings & b) associated flipped homework (See next date for reading assignment)

*'''9-3''' ''How People Learn; Instructional complexities; Project topic brainstorming''
**Read Ch2. How Do People Learn from E-Courses (20 pages) [[Media:E-Learning-Ch2.pdf|You can get the chapter here this '''last''' time!]] ''Please order the book now if you have not!''
**Read Koedinger et al. (2013) paper [[Media:Koedinger-Science-2013.pdf|Instructional complexities]]
***Do the quiz for Chapter 2 (Quiz 2).
***On Blackboard do a "Discussion Board" post for Instructional complexities paper within the forum titled "Instructional Complexity Reading Posts for 10-3"
***Slides for this chapter are [[Media:L02-how-people-learn+instr-complexity.pptx|here]].
**Class activity: Promises & pitfalls review of e-learning examples
***BRING a print-out of an e-learning example to class
**Class activity: Project idea discussion
***Come prepared with a ''preliminary project idea'' -- post one or more project ideas in Discussion Board forum "Project idea feedback & partner solicitation"
**For next time:
***a) Do the readings & b) associated flipped homework

=====Instructional Goals and Assessment 9-8 to 9-17=====

*9-8 Determining instructional goals; KLI KCs; Bloom's taxonomy
**Reading: [[Media:CogInstCarver12.pdf|Carver paper]]
**Reading: KLI sections 1-3 [[Media:KLI-Framework-KoedingerCorbettPerfetti2012.pdf|KLI Framework paper]] (we will discuss other sections later)
**Additional Reading: [[Media:Krathwohl_Bloom's taxonomy revised.pdf|Bloom's taxonomy revised]]
***Do Discussion Board post on Carver paper.
***Do quiz for the KLI reading.
**Class activity: Review Project ideas and step 1 write-up requirements
**Class activity: KC type in e-learning examples

*9-10 Writing assessments to meet goals; Evidence-centered design
**Reading: [https://www.cmu.edu/teaching/assessment/index.html Assessment design--Eberly Center web pages on Assess Teaching and Learning] Read: Basics, Prior Knowledge, Assessing Learning, Examples and Tools (Skip: Assessing Teaching, Assessing Programs, History at Carnegie Mellon)
**Reading: [[Media:Evidence-centered-design-2003.pdf|Evidence-centered design]]
***Do TWO discussion board posts on readings.
**Class activity: Goal setting & designing assessments that provide evidence of goal achievement

*9-15 Why data toward goal setting improves design
**Reading: [[Media:Feldon_Timmerman_etal_2010.pdf | Feldon paper]]
**At least skim: [[Media:Clark_CTA_In_Healthcare_Chapter_2012.pdf | Clark_CTA_In_Healthcare_Chapter_2012]]
**Reading: [[Media:Contextual Inquiry Principles.pdf|Contextual inquiry principles]]
***Do quiz for the Feldon reading.
***Do ONE discussion board post.
**Class activity: Interviewing practice

*9-17 Online assessment; Practice e-assessment implementation
**Reading: Read the documentation for two online assessment authoring tools of your choosing
***Discussion board posts on pros and cons of the tools you read about
**Class activity: BRING YOUR LAPTOP and be prepared to use an online assessment development tool
**DUE: P1: Context & Initial Resources '''(Submit as a Word document.)'''
**Assignment: P2 is due Oct 1

=====Cognitive Task Analysis (CTA) 9-22 to 10-1=====
*9-22 Empirical CTA: Structured Interviews
**Reading: [[Media:Clarketal2007-CTAchapter proof-1.pdf | Clark et al., 2007 Chapter proof]]
**Reading: [[Media:Contextual Inquiry Practice.pdf|Contextual inquiry in practice]]
**Reading: [[Media:Koh .pdf|CTA interview strategies]] (a paper by a former METALS student!)
**Other Readings: Working Minds: A practitioner's Guide to Cognitive Task Analysis, [[Media:WorkingMinds_Chap_1.pdf|Ch 1]] and [[Media:WorkingMinds_Ch_2.pdf|Ch 2]]; [[Media:Working Minds-Ch5-interviews.pdf |Working Minds Ch 5]];
***Do posts for readings.

*9-24 Think Alouds & Rational CTA
**Reading: [[Media:Lovett98.pdf|Lovett paper]] and [[Media:Gomoll-90.pdf|Gomoll paper]] -- [[Media:L05_Think_Aloud_2014-Lovett-Gomoll-for_syllabus.pptx|Slides]]
**Reading: [[Media:Zhu&Simon-1987.pdf|Zhu & Simon paper]] -- [[Media:L06-ELDP-Rational-CTA-2014-b-for_syllabus.pptx|Slides]]
***Do quizzes for the readings.

*9-29 Quantitative Cognitive Task Analysis: Difficulty Factors Assessment
**Reading: [[Media:CogSci97-Heffernan-distrib.pdf‎ | Heffernan paper]] -- [[Media:L08-ELDP-CTA-DFA-2014-Heffernan_forsyllabus.pptx|Slides]]
**Come with an initial draft of project step 2.
***Do quiz for Heffernan paper.

*10-1 Quantitative CTA via Data Mining; CTA to improve instructional design
**Class activity: Work on P3. Analyzing your data
**Reading: [[Media:Koedinger-et-al-aied2013.pdf | e-learning data to improvement]] (10 pages)
***Do discussion board post(s) for e-learning data to improvement paper
**DUE: P2: Identifying Goals & Online Assessment Creation
**Assignment: P3 is due Oct 15

=====Multimedia Principles 10-6 to 10-22=====
*10-6 Evidence-based practice; KLI Learning & Instructional Events
**Reading: Clark & Mayer book Ch3.Evidence-based practice (18 pages)
**Reading: KLI paper sectionts 4-5 (12 pages)
*** Do quizzes for the readings.
**Class activity: Principles present in e-learning examples

*10-8 Multimedia Principle; Contiguity Principle; Practice applying
**Reading: 4.Multi-media Principle (24 pages) [[Media:L07-ELDP-Multimedia-2014_for_syllabus.pptx|Slides]]
**Reading: 5.Contiguity Principle (24 pages)
*** Do quizzes for the readings.
**Optional readings & slides:
*** Systematic Thinking Fostered by Illustrations in Scientific Text [[Media:Mayer_89.pdf|Paper]] [[Media:Mayer_89.pptx|Slides]]
*** Multimedia-Supported Metaphors for Meaning Making in Mathematics [[Media:Moreno & Mayer_99.pdf|Paper]] [[Media:Morena-Mayer-1999.pptx|Slides]]

*10-13 Modality Principle & Redundancy Principle; Practice applying
**Class activity: Work on P3. How will you collect data?
**Reading: 6.Modality Principle (18 pages)
**Reading: 7.Redundancy Principle (18 pages)
***Do quizzes for the readings.

*10-15 Coherence Principle & Personalization Principle; Practice applying
**Reading: 8.Coherence Principle (28 pages)
**Reading: 9.Personalization Principle (26 pages)
***Do quizzes for the readings.
**Due: P3: Cognitive Task Analysis & Cognitive Model
**Assignment: P4 is due Oct 29

*10-20 Flex topic; Midterm Review

*10-22 Midterm exam '''Bring laptop to class'''
**Optional Readings:
***[http://www.articulate.com/rapid-elearning/visual-graphic-design/ Visual & Graphic Design for e-learning blog]
***[[Media:Kirschner-Merrienboer-2013.pdf | Do Learners Really Know Best? Urban Legends in Education]]

=====Learning By Doing Principles 10-27 to 11-24 =====

*10-27 KLI & Selecting appropriate instructional principles
**Reading: KLI sections 6-7
***Do quizzes for the reading.
**Assignment: P4 is due 10-29

*10-29 Segmenting and Pretraining; Leveraging Examples in E-Learning
**Reading: 10.Segmenting and Pretraining (18 pages)
**Reading: 11.Leveraging Examples in E-Learning (28 pages)
***Do quizzes for the readings.
**DUE: P4: Initial Instructional Design
**Assignment: P5 is due 11-19

*11-3 Does Practice Make Perfect; Who’s in Control?
**Reading: 12.Does Practice Make Perfect (28 pages)
**Reading: 14.Who’s in Control? (30 pages)
***Do quizzes for the readings.

*11-5 E-Learning to Build Problem Solving Skill; Simulations and Games
**Reading: 15.E-Learning to Build Problem Solving Skill (30 pages)
**Reading: 16.Simulations and Games (32 pages)
***Do quizzes for the readings.

*11-10 Applying the Guidelines; KLI Review
**Reading: 17.Applying the Guidelines (24 pages)
***Do quizzes for the reading.

*11-12 [Guest topic: Options CSCL, Cognitive Mastery, Hint Factory, CTAT?]

*11-17 Peer review of instructional design

*11-19 In vivo experimentation; A/B Testing
**DUE: P5: Instructional Design Prototyping & Testing
**Assignment: P6 is due 12-1

*11-24Flex topic; Presentation & Report Preparation

*11-26 Thanksgiving, no class

=====Final & Project Presentations 12-1 to 12-10=====

*12-1 Final Exam '''Bring laptop to class'''
**If needed: Final Exam Make-up - Thurs Dec 17, 1-4pm in GHC 5222
** DUE: P6: Experimental Design
**Assignment: Final Project is due 12-14. It should include the reflection statement (see the project assignment handout).

*12-3 Project Presentations
*12-8 Project Presentations
*12-10 Project Presentations (we need to reschedule this, perhaps to 12-11)

=====Final Project Due on 12-14=====

File:Morena-Mayer-1999.pptx

2015-10-01T10:54:30Z

Koedinger:

File:Moreno & Mayer 99.pdf

2015-10-01T10:52:56Z

Koedinger: