Theory Wiki - User contributions [en]

Assistance dilemma

2011-08-26T09:25:03Z

Woolerystixmaker:

[[Category:Glossary]]
[[Category:PSLC General]]

A fundamental problem of instructional engineering is: when should instruction provide students with [[assistance]] and when should it withhold assistance? We call this problem the ''assistance dilemma'' (Koedinger & Aleven, in press). The dilemma emerges because there are complementary benefits and costs of providing higher vs. lower levels of instructional assistance. Lower assistance challenges students to generate and construct knowledge on their own, but may leave them floundering, frustrated and wasting time. Higher assistance can provide students with information they will not generate on their own, but may reduce engagement or prevent formation of lasting memories.

In fact, the assistance dilemma is a central battleground of the education wars, with one side advocating more direct instruction and drill of basic skills, that is, higher assistance, and other side advocating more student initiative, construction, discovery, and learning by doing, that is, lower assistance. Our view is that the assistance dilemma will not be resolved by determining which side is right, but by specifying dimensions of assistance and, ultimately, identifying and fitting parameters along these dimensions to determine the optimal level of assistance given the instructional goal and the students' knowledge state relative to that goal.

Koedinger, K. R., & Aleven V. (in press). Exploring the assistance dilemma in experiments with Cognitive Tutors. ''Educational Psychology Review''.

[http://custom-essay-writing-service.org/index.php custom writing]

Baker - Building Generalizable Fine-grained Detectors

2011-08-26T09:24:51Z

Woolerystixmaker:

== Building Generalizable Fine-grained Detectors ==

=== Summary Table ===
====Study 1====
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
| '''PIs''' || Ryan Baker, Vincent Aleven
|-
| '''Other Contributors''' || Sidney D'Mello (Consultant, University of Memphis), Ma. Mercedes T. Rodrigo (Consultant, Ateneo de Manila University)

|-
| '''Study Start Date''' || February, 2010
|-
| '''Study End Date''' || February, 2011
|-
| '''LearnLab Site''' || TBD
|-
| '''LearnLab Course''' || Algebra, Geometry, Chemistry, MathTutor, ScienceAssistments
|-
| '''Number of Students''' || 78 so far; total TBD
|-
| '''Total Participant Hours''' || 444 so far; total TBD
|-
| '''Data available in DataShop''' || [https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=431 Dataset: CMU VlabHomeworks F2010]<br>
[https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=448 Dataset: Affect Detectors and Questionnaires Greenville 2010-11]<br>
* '''Pre/Post Test Score Data:''' TBD
* '''Paper or Online Tests:''' TBD
* '''Scanned Paper Tests:''' TBD
* '''Blank Tests:''' TBD
* '''Answer Keys: ''' TBD
|}

=== Abstract ===
This project, joint between M&M and CMDM, will create a set of fine-grained detectors of affect and M&M behaviors. These detectors will be usable by future projects in these two thrusts to study the impact of learning interventions on these dimensions of students’ learning experiences, and to study the inter-relationships between these constructs and other key PSLC constructs (such as measures of robust learning, and motivational questionnaire data). It will be possible to apply these detectors retrospectively to existing PSLC data in [[DataShop]], in order to re-interpret prior work in the light of relevant evidence on students’ affect and M&M behaviors.

=== Background & Significance ===

=== Glossary ===

[[Metacognition and Motivation]]

[[Computational Modeling and Data Mining]]

[[Gaming the system]]

[[Off-Task Behavior]]

[[Affect]]

[[Frustration]]

[[Boredom]]

[[Flow]]

[[Engaged Concentration]]

=== Hypotheses ===

H1: We hypothesize that it will be possible to develop reasonably accurate detectors of student affect for four LearnLabs, that detect affect using only the data from the interaction between the student and the keyboard/mouse.

H2: We hypothesize that models of behaviors such as gaming the system, and off-task behavior, in combination with models of affect/behavior dynamics, can make affect detectors more accurate.

H3: We hypothesize that models created using data from three LearnLabs will perform significantly better than chance in data from a fourth LearnLab, with no re-training (or limited EM-based modification that requires no new labeled data).

H4: We hypothesize that these affect models will become a valuable component of future research in the M&M and CMDM thrusts.

=== Research Process ===

We will develop detectors of the M&M (metacognitive & motivational) behaviors of gaming the system, off-task behavior, proper help use, on-task conversation, help avoidance and self-explanation without scaffolding. This set of behaviors has already been effectively detected in mathematics LearnLabs. We will model the dynamics between these behaviors and student affect (following on work in the PSLC and at Memphis), in order to be able to leverage these detectors to create detectors of the affective states of engaged concentration, boredom, confusion, and frustration (the dynamics models will enable us to set Bayesian priors for how likely an affective state is at a given time).

These detectors will be developed for multiple LearnLabs, and the generalizability of detectors across LearnLabs will be one of the focuses of study during this project. We anticipate developing detectors for Algebra and Geometry, the Chemistry Virtual Lab, MathTutor, and Science ASSISTments. Each of these learning environments presents a context where complex learning occurs, fine-grained interaction behavior is logged, and the outputs of the detectors will provide leverage on a number of research questions of interest.

“Ground truth” for the M&M behavior categories will be established through quantitative field observations. “Ground truth” for the affect categories will be established by field observations and infrequent pop-up questions. Work will be conducted to increase the reliability of quantitative field observations of affect to a standard considered appropriate by psychology journals, through repeated coding and discussion sessions and the development of a detailed coding manual based on prior work to code affect in field settings and work to code emotions from facial expressions.

Models will be developed solely using distilled log file data of the sort currently collected in [[DataShop]] (more sophisticated sensors will NOT be included in this project). The models will be built with a combination of machine learning, and knowledge engineering (specifically, through leveraging and adapting existing knowledge engineered models such as Aleven et al’s help-seeking model and Shih et al’s self-explanation model). Generalization of models across learning environments will involve expectation maximization to adapt models to new data sets, and/or leveraging the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features. We will first develop models for individual learning environments and then extend them across environments.

=== Research Plan ===

1. Develop software for conducting field observations (cf. Baker et al, 2004) with PDAs and synchronizing with [[DataShop]] data -- software development completed, as of Aug 2010 synchronization verification in progress

2. Study and improve quantitative field coding of student affect states

* The Research Associate and Assistant will conduct multiple coding and discussion sessions with the PI, and develop a detailed coding manual (including some video examples)

3. Collect training data (months 4-7) -- as of Aug 2010 first data set collected, other data collection in progress

* Starting first in one LearnLab and rolling across LearnLabs, so that we have all the data for one LearnLab first. Collecting data on all constructs at once. Then the programmer/PI can start developing detectors for constructs in first LearnLab, while the RAs keep collecting more data in the second and subsequent LearnLabs
* Quantitative field observations (cf. Baker et al, 2004)

4. Develop detectors (months 5-8)

* Utilizing combination of existing data mining tools and code previously used by Baker to create Latent Response Model-based detectors of [[Gaming the System]] and [[Off-Task Behavior]]

* Develop and leverage behavior-affect temporal dynamics models (cf. D’Mello et al, 2007; Baker, Rodrigo, & Xolocotzin, 2007) to create priors for predicting affect

* Use log data to predict field observations, student responses

* Student-level cross-validation used for assessing goodness of detectors

5. Develop meta-detectors (months 9-12)

* Use expectation maximization to adapt models to new data sets

* Leverage the CTLVS1 taxonomy to develop meta-models that relate prediction features to design features

* Cross-validation at grain-size of transfer between units or corresponding (within each LearnLab) to validate appropriateness for whole LearnLab

* Test goodness of models when {train on 3 tutors, transfer to tutor #4} to evaluate effectiveness for entirely new tutors

=== Independent Variables ===

n/a (see Research Plan)

=== Dependent Variables ===

n/a (see Research Plan)

=== Affective States and M&M Behaviors to be Modeled ===

Affective States:
* Engaged Concentration (a subset of [[Flow]]) (cf. Baker et al, 2010)
* Boredom (Kapoor, Burleson, & Picard, 2007)
* Frustration (Kapoor, Burleson, & Picard, 2007)

M&M Behaviors:

* [[Gaming the system]] (Baker et al, 2004)
* [[Off-Task Behavior]] (Baker, 2007)
* Proper Help Use (Aleven et al, 2006)
* On-Task Conversation
* [[Help Avoidance]] (Aleven et al, 2006)
* [[Self-Explanation]] without scaffolding (Shih et al, 2008)

=== Planned Studites ===

In 2010, data will be collected in the Algebra, Geometry, Chemistry, MathTutor, and Science ASSISTments.

=== Explanation ===
=== Further Information ===
=== Connections ===

=== Annotated Bibliography ===
=== References ===

Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a Cognitive Tutor. International Journal of Artificial Intelligence and Education, 16, 101-128.

Baker, R.S.J.d. (2007) Modeling and Understanding Students' Off-Task Behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068.

Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390.

Baker, R.S.J.d., Rodrigo, M.M.T., Xolocotzin, U.E. (2007) The Dynamics of Affective Transitions in Simulation Problem-Solving Environments. Proceedings of the Second International Conference on Affective Computing and Intelligent Interaction.

D'Mello, S. K., Picard, R. W., and Graesser, A. C. (2007) Towards an Affect-Sensitive AutoTutor. Special issue on Intelligent Educational Systems – IEEE Intelligent Systems, 22(4), 53-61.

Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International Journal of Human-Computer Studies, 65, 724-736.

Shih, B., Koedinger, K., and Scheines, R. (2008) A Response Time Model for Bottom-Out Hints as Worked Examples. Proceedings of the 1st International Conference on Educational Data Mining, 117-126.

=== Future Plans ===
[http://custom-essay.ws/index.php essay papers]

Application of SimStudent for Error Analysis

2011-08-26T09:24:27Z

Woolerystixmaker:

==Towards a theory of learning errors==

===Personnel===

*PI: Noboru Matsuda
*Key Faculty: William W. Cohen, Kenneth R. Koedinger

===Abstract===

The purpose of this project is to study how students ''fail to learn'' correct knowledge components when studying from examples and make (typical) errors by applying such incorrect knowledge components later when solving problems. We utilize a computational model of learning, called [http://www.SimStudent.org SimStudent] that learns cognitive skills inductively from examples either by passively reviewing worked-out examples or by actively engaged in tutored problem-solving.

We are particularly interested in studying how the differences in prior knowledge affect the nature and rate of learning. We hypothesize that when students rely on shallow, domain general features (which we call "weak" features) as opposed to deep, more domain specific features ("strong" features), then students would more likely to make induction errors.

To test this hypothesis, we control SimStudent's prior knowledge to study how and when erroneous skills are learned by analyze learning outcomes (both the process of learning and the performance on the post-test).

===Overview of SimStudent===

A fundamental technology used for SimStudent is called Inductive Logic Programming (Muggleton, 1999) as an application for programming by demonstration (Cypher, 1993). Prior to learning, SimStudent is given a set of ''feature predicates'' and ''operators'' as prior knowledge.

Feature predicate is a Boolean function to test an existence of a certain feature. For example, isPolynomial("3x+1") returns true, but isConstantTerm("3x") returns false. An operators, on the other hand, is a more generic function to manipulate various form of objects involved in a target task. For example, addTerm("3x", "2x") returns "5x" and getCoefficient("-4y") returns "-4."

To learn cognitive skills, SimStudent generalizes ''examples'' of each individual skill applications. There are two types of examples necessary to given to SimStudent: (1) positive examples that show when to apply a particular skill, and (2) negative examples that show when ''not'' to apply a particular skill.

Positive examples are acquired either from (1) steps demonstrated in worked-out examples, (2) steps demonstrated as a hint during tutoring, and (3) steps performed correctly by SimStudent itself during tutoring. In either case, a context of a skill application (i.e., a problem status) is stored as a positive examples for that particular skill.

Negative examples are acquired either when (1) a positive example is generated, or (2) SimStudent made an error during tutoring. When a positive example is made for a certain skill, say S, the example also becomes negative examples for all other skills than S. Such an example is called ''implicit negative example.'' An implicit negative example becomes a positive example if the corresponding skill is applied in the specified situation.

Given a set of positive and negative examples for a skill, SimStudent generates a hypothesis (in the form of production rule) representing when and how to apply the skill. The hypothesis is generated so that it applies to all positive examples and none of the negative examples.

===Background and Significance===

There are a number of models of student errors proposed so far (Brown & Burton, 1978; Langley & Ohlsson, 1984; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Weber, 1996; Young & O'Shea, 1981). Our effort builds on the past works by exploring how differences in prior knowledge affect the nature of the incorrect skills acquired and the errors derived. We are particularly interested in errors that are made by applying incorrect skills, and our computational model explains the processes of learning such incorrect skills as incorrect induction from examples.

We hypothesize that incorrect generalizations are more likely when students have weaker, more general prior knowledge for encoding incoming information. This knowledge is typically perceptually grounded and is in contrast to deeper or more abstract encoding knowledge. An example of such perceptually grounded prior knowledge is to recognize 3 in x/3 simply as a number instead of as a denominator. Such an interpretation might lead students to learn an inappropriate generalization such as "multiply both sides by a number in the left hand side of the equation" after observing x/3=5 gets x=15. If this generalization gets applied to an equation like 4x=2, the error of multiplying both sides by 4 is produced.

We call this type of perceptually grounded prior knowledge "weak" prior knowledge in a similar sense as Newell and Simon’s weak reasoning methods (1972). Weak knowledge can apply across domains and can yield successful results prior to domain-specific instruction. However, in contrast to "strong" domain-specific knowledge, weak knowledge is more likely to lead to incorrect conclusions.

In general, a particular example can be modeled both with weak and strong operators. For example, suppose a step x/3=5 gets demonstrated to "multiply by 3." Such step can be explained by a strong operator getDenominator(x/3), which returns a denominator of a given fraction term and multiply that number to both sides. On the other hand, the same step can be explained by a weak operator
getNumberStr(x/3), which returns the left-most number in a given expression. In this context, the operator getNumberStr() is considered to be weaker than the operator getDemonimator(), because a production rule with getNumberStr() explains broader errors. For example, imagine how we could model the error schema for "multiply by A." This error schema can be modeled with getNumberString() and multiply() – get a number and multiply both sides by that number. Without the weak operator, we need to have different (disjunctive) production rules to model the same error schema for different problem schemata – getNumerator() for A/v=C and getCoefficient() for Av=C.

===Human Students Error Analysis===

===Research Question===

How do the differences in prior knowledge affect the type and rate of learning errors? Especially, does "weak" prior knowledge foster more induction errors than "strong" prior knowledge, and if so, to what extent does such "weak" prior-knowledge learning account for errors that (human) students commonly make?

===Hypothesis===

Since "weak" prior knowledge applies broader context than "strong" prior knowledge, when given "weak" prior knowledge SimStudent would learn overly general rules that make more human-like errors.

===Study Variables===

====Independent Variable====

'''Prior knowledge''': implemented as "operator" and "feature predicates" for SimStudent.

====Dependent Variables====

'''Step score''': For a quantitative assessment, we computed a ''step score'' for each step in the test problems as follows: 0 if there is no correct rule application made, otherwise it is a ratio of the number of correct rule applications to the number of all rule applications allowing SimStudent to show all possible rule applications on the step.

'''Error prediction''': For a qualitative assessment, we are particularly interested in errors made by applying learned rules as well as the accuracy of prediction. Given a step ''S'' performed by a human student at an intermediate state ''N'', SimStudent is asked to compute a conflict set on ''N''. Rule application R''i'' (''i'' = 1, …, ''n'') is coded as follows:

: True Positive: R''i'' yields the same step as ''S'', and ''S'' is a correct step.
: False Positive: R''i'' yields a correct step that is not same as ''S'' (''S'' may be incorrect).
: False Negative: R''i'' yields an incorrect step that is not same as ''S'' (''S'' may be correct).
: True Negative: R''i'' yields the same step as ''S'' and ''S'' is an incorrect step.

Error prediction is computed as True Negative / (True Negative + False Negative) to understand how well SimStudent predicted human-like errors.

===Findings===

====Learning Curve====

Figure 1 shows average step score, aggregated across the test problems and student conditions. The X-axis shows the number of training iterations.

The Weak-PK and Strong-PK conditions had similar success rates on test problems after the first 8 training problems. After that, the performance of the two conditions began to diverge. On the final test after 20 training problems, the Strong-PK condition was 82% correct while the Weak-PK was 66%, a large and statistically significant difference (t = 4.00, p < .001).

A simple fit to power law functions to the learning curves (converting success rate to log-odds) showed that the slope (or rate) of the Weak-PK learning curve (.78) is smaller (or slower) than that of the Strong-PK learning curve (.82). We then subtracted the two functions in their log-log form and verified in a linear regression analysis that the coefficient of the number of training problems (which predicts the difference in rate) is significantly greater than 0 (p < .05).

[[Image:NM-LearningCurve.jpg]]

Figure 1: Average step score after each of the 20 training problems for SimStudents with either strong or weak prior knowledge.

====Error Prediction====

Figure 2 shows a number of true negative predictions made on the test problems for each of the training iterations.

Surprisingly, the Weak PK condition did make as many as 22 human-like errors on the 11 test problems. On the other hand, the Strong PK condition hardly made human-like errors.

[[Image:NM-Num-TN-Prediction.jpg]]

Figure 2: Number of True Negative predictions, which are the same errors made both by SimStudent and human students on the same step in the test problems.

===Publications===

*Matsuda, N., Lee, A., Cohen, W. W., & Koedinger, K. R. (2009; to appear). A Computational Model of How Learner Errors Arise from Weak Prior Knowledge. In Conference of the Cognitive Science Society.

===References===

*Booth, J. L., & Koedinger, K. R. (2008). Key misconceptions in algebraic problem solving. In B. C. Love, K. McRae & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 571-576). Austin, TX: Cognitive Science Society.

*Muggleton, S. (1999). Inductive Logic Programming: Issues, results and the challenge of Learning Language in Logic. Artificial Intelligence, 114(1-2), 283-296.

*Cypher, A. (Ed.). (1993). Watch what I do: Programming by Demonstration. Cambridge, MA: MIT Press.

[http://custom-essay-writing-service.org/index.php custom essay writing]

Analogical comparison

2011-08-26T09:23:30Z

Woolerystixmaker:

Analogical comparison operates through aligning and mapping two example problem representations to one another and then extracting their commonalities (Gentner, 1983; Gick & Holyoak, 1983; Hummel & Holyoak, 2003). This process discards the elements of the knowledge representation that do not overlap between two examples but preserves the common elements. The resulting knowledge organization typically consists of fewer superficial similarities (than the examples) but retains the deep causal structure of the problems.

===References===
*Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy, ''Cognitive Science, 7'', 155-170.
*Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. ''Cognitive Psychology, 15'', 1-38.
*Hummel, J. E., & Holyoak, K. J. (2003). A symbolic-connectionist theory of relational inference and generalization. ''Psychological Review, 110'', 220-264.

[[Category: Learning Processes]]
[[Category: Glossary]]
[[Category: Coordinative Learning]]

[http://custom-essay-writing-service.org/index.php custom writing]

[http://custom-essay-writing-service.org/index.php essay writing service]