Student Uncertainty - Revision history

Forbesk: /* Findings */

2008-03-12T18:13:06Z

Findings

← Older revision		Revision as of 18:13, 12 March 2008
Line 88:		Line 88:
	Based on these results, we hypothesize that the tutoring treatment was too short to yield significant differences between conditions in learning, as measured by our pretest and posttest.		Based on these results, we hypothesize that the tutoring treatment was too short to yield significant differences between conditions in learning, as measured by our pretest and posttest.

	We used the isomorphic second problem as an additional test of how the uncertainty adaptation in the first problem impacted student answers to the isomorphic questions in the test problem. The non-adaptive system was used in all conditions for the test problem, so that all students received the same "test". We hypothesized that the conditions would differ with respect to various dialogue-based performance metrics measured in the test problem, depending on whether or not the uncertainty adaptation was received in the first problem. Our performance metrics include correctness, uncertainty, and learning impasse severities. Our results are described in detail in Forbes-Riley et al. (2008). Among other results, we found that correct+uncertain answers are more likely to become incorrect in the second problem if the uncertainty adaptation is not received during the tutoring treatment, but only in the random control condition are these answers also more likely to become nonuncertain.		We used the isomorphic second problem as an additional test of how the uncertainty adaptation in the first problem impacted student answers to the isomorphic questions in the test problem. The non-adaptive system was used in all conditions for the test problem, so that all students received the same "test". We hypothesized that the conditions would differ with respect to various dialogue-based performance metrics measured in the test problem, depending on whether or not the uncertainty adaptation was received in the first problem. Our performance metrics include correctness, uncertainty, and learning impasse severities. Our results are described in detail in Forbes-Riley et al. (2008). Among other results, we found that correct+uncertain answers are more likely to become incorrect in the second problem if the uncertainty adaptation is not received during the tutoring treatment (p < 1.0), but only in the random control condition are these answers also more likely to become nonuncertain (p < 0.05).

	=== Explanation ===		=== Explanation ===

Forbesk: /* Findings */

2008-03-12T18:10:28Z

Findings

← Older revision		Revision as of 18:10, 12 March 2008
Line 88:		Line 88:
	Based on these results, we hypothesize that the tutoring treatment was too short to yield significant differences between conditions in learning, as measured by our pretest and posttest.		Based on these results, we hypothesize that the tutoring treatment was too short to yield significant differences between conditions in learning, as measured by our pretest and posttest.

	We used the isomorphic second problem as an additional test of how the uncertainty adaptation in the first problem impacted student answers to the isomorphic questions in the test problem. The ~~{\it~~ non-adaptive} system was used in all conditions for the test problem, so that all students received the same "test". We hypothesized that the conditions would differ with respect to various dialogue-based performance metrics measured in the test problem, depending on whether or not the uncertainty adaptation was received in the first problem. ~~See~~ Forbes-Riley et al. (2008) ~~for further details~~.		We used the isomorphic second problem as an additional test of how the uncertainty adaptation in the first problem impacted student answers to the isomorphic questions in the test problem. The non-adaptive system was used in all conditions for the test problem, so that all students received the same "test". We hypothesized that the conditions would differ with respect to various dialogue-based performance metrics measured in the test problem, depending on whether or not the uncertainty adaptation was received in the first problem. Our performance metrics include correctness, uncertainty, and learning impasse severities. Our results are described in detail in Forbes-Riley et al. (2008). Among other results, we found that correct+uncertain answers are more likely to become incorrect in the second problem if the uncertainty adaptation is not received during the tutoring treatment, but only in the random control condition are these answers also more likely to become nonuncertain.

	=== Explanation ===		=== Explanation ===

Forbesk: /* Further Information */

2008-03-12T18:05:27Z

Further Information

← Older revision		Revision as of 18:05, 12 March 2008
Line 112:		Line 112:
	Forbes-Riley, Kate and Diane Litman. 2004. Predicting emotion in spoken dialogue from multiple knowledge sources. In Proceedings of the Human Language Technology Conference: 4th Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL), pages 201-208, Boston, MA. (This paper describes some of our prior work on automatically predicting student emotions in human-human tutoring.)		Forbes-Riley, Kate and Diane Litman. 2004. Predicting emotion in spoken dialogue from multiple knowledge sources. In Proceedings of the Human Language Technology Conference: 4th Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL), pages 201-208, Boston, MA. (This paper describes some of our prior work on automatically predicting student emotions in human-human tutoring.)

	Kate Forbes-Riley, Mihai Rotaru, Diane J. Litman and Joel Tetreault, Exploring Affect-Context Dependencies for Adaptive System Development~~.2007~~. Proceedings Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Rochester, NY, April.		Kate Forbes-Riley, Mihai Rotaru, Diane J. Litman and Joel Tetreault. 2007. Exploring Affect-Context Dependencies for Adaptive System Development. Proceedings Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Rochester, NY, April. (This paper explores analyses leading to plausible automatic adaptations to student uncertainty.)
	~~[http://www~~.~~cs.pitt.edu/~litman/frtl-naacl07-final.pdf]~~

	Hausmann, Robert and Michelene Chi. 2002. Can a computer interface support self-explaining? The International Journal of Cognitive Technology, 7(1):4-14. (This paper shows that spontaneous self-explanation occurs more frequently in spoken tutoring then in text-based tutoring.)		Hausmann, Robert and Michelene Chi. 2002. Can a computer interface support self-explaining? The International Journal of Cognitive Technology, 7(1):4-14. (This paper shows that spontaneous self-explanation occurs more frequently in spoken tutoring then in text-based tutoring.)

			Litman, Diane. April 2008. Detecting and Adapting to Student Uncertainty in a Spoken Tutorial Dialogue System. Invited Talk at
			Affective Language in Human and Machine Symposium, AISB Convention, Aberdeen, Scotland. (This is an invited talk about the experiment described in this wiki page, as well as another experiment.)

	Litman, Diane and Kate Forbes-Riley. 2004a. Annotating student emotional states in spoken tutoring dialogues. In Proceedings of 5th SIGdial Workshop on Discourse and Dialogue (SIGdial), pages 144-153, Boston, MA, April. (This paper describes some of our prior work on annotating student uncertainty and other emotions.)		Litman, Diane and Kate Forbes-Riley. 2004a. Annotating student emotional states in spoken tutoring dialogues. In Proceedings of 5th SIGdial Workshop on Discourse and Dialogue (SIGdial), pages 144-153, Boston, MA, April. (This paper describes some of our prior work on annotating student uncertainty and other emotions.)

Forbesk: /* Findings */

2008-03-12T15:08:23Z

Findings

← Older revision		Revision as of 15:08, 12 March 2008
Line 87:		Line 87:

	Based on these results, we hypothesize that the tutoring treatment was too short to yield significant differences between conditions in learning, as measured by our pretest and posttest.		Based on these results, we hypothesize that the tutoring treatment was too short to yield significant differences between conditions in learning, as measured by our pretest and posttest.

			We used the isomorphic second problem as an additional test of how the uncertainty adaptation in the first problem impacted student answers to the isomorphic questions in the test problem. The {\it non-adaptive} system was used in all conditions for the test problem, so that all students received the same "test". We hypothesized that the conditions would differ with respect to various dialogue-based performance metrics measured in the test problem, depending on whether or not the uncertainty adaptation was received in the first problem. See Forbes-Riley et al. (2008) for further details.

	=== Explanation ===		=== Explanation ===

Forbesk: /* Findings */

2008-03-12T15:03:43Z

Findings

← Older revision		Revision as of 15:03, 12 March 2008
Line 85:		Line 85:

	However, in a two-way ANOVA with condition by repeated test measures design, there was a significant main effect for test phase (F(1,57) = 34.88, p = 0.000, MSe = 0.032), indicating that students in all conditions learned a significant amount during tutoring. However, there was no significant interaction effect between condition and test phase, indicating that how much students learned was not dependent on condition.		However, in a two-way ANOVA with condition by repeated test measures design, there was a significant main effect for test phase (F(1,57) = 34.88, p = 0.000, MSe = 0.032), indicating that students in all conditions learned a significant amount during tutoring. However, there was no significant interaction effect between condition and test phase, indicating that how much students learned was not dependent on condition.

			Based on these results, we hypothesize that the tutoring treatment was too short to yield significant differences between conditions in learning, as measured by our pretest and posttest.

	=== Explanation ===		=== Explanation ===

Forbesk: /* Expected Findings */

2008-03-12T15:01:45Z

Expected Findings

← Older revision		Revision as of 15:01, 12 March 2008
Line 80:		Line 80:
	We will also investigate the proportion of correct answers on the second physics problem, across conditions (Note: This does not measure learning as a consequence of the instructional conditions as these instructional variations are back in place during this phase).		We will also investigate the proportion of correct answers on the second physics problem, across conditions (Note: This does not measure learning as a consequence of the instructional conditions as these instructional variations are back in place during this phase).

	=== ~~Expected~~ Findings ===		=== Findings ===

	~~The~~ response to uncertain student answers in the experimental condition ~~will~~ yield significantly higher learning gains on the post-test than either not responding (first control condition), or treating a random subset of correct student answers as if they were incorrect (second control condition).		We hypothesized that the response to uncertain student answers in the experimental condition would yield significantly higher learning gains on the post-test than either not responding (first control condition), or treating a random subset of correct student answers as if they were incorrect (second control condition).

			However, in a two-way ANOVA with condition by repeated test measures design, there was a significant main effect for test phase (F(1,57) = 34.88, p = 0.000, MSe = 0.032), indicating that students in all conditions learned a significant amount during tutoring. However, there was no significant interaction effect between condition and test phase, indicating that how much students learned was not dependent on condition.

	=== Explanation ===		=== Explanation ===

Forbesk: /* Further Information */

2008-03-12T14:51:10Z

Further Information

← Older revision		Revision as of 14:51, 12 March 2008
Line 99:		Line 99:

	Craig, Scotty, Arthur Graesser, Jeremiah Sullins, and Barry Gholson. 2004. Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3):241-250. (This paper describes correlations between uncertainty and learning in tutoring.)		Craig, Scotty, Arthur Graesser, Jeremiah Sullins, and Barry Gholson. 2004. Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3):241-250. (This paper describes correlations between uncertainty and learning in tutoring.)

			Forbes-Riley, Kate, Diane Litman, and Mihai Rotaru. 2008. Responding to Student Uncertainty during Computer Tutoring: A Preliminary Evaluation. Proceedings 9th International Conference on Intelligent Tutoring Systems (ITS), Montreal, Canada, June. (This paper describes in detail the results of the experiment overviewed in this wiki page.)

			Forbes-Riley, Kate, Diane Litman, Scott Silliman, and Amruta Purandare. 2008. Uncertainty Corpus: Resource to Study User Affect in Complex Spoken Dialogue Systems. Proceedings 6th Language Resources and Evaluation Conference (LREC 2008), Marrakech, Morocco, May-June. (This paper describes in detail the corpus collected from the experiment overviewed in this wiki page.)

	Forbes-Riley, Kate and Diane Litman. 2004. Predicting emotion in spoken dialogue from multiple knowledge sources. In Proceedings of the Human Language Technology Conference: 4th Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL), pages 201-208, Boston, MA. (This paper describes some of our prior work on automatically predicting student emotions in human-human tutoring.)		Forbes-Riley, Kate and Diane Litman. 2004. Predicting emotion in spoken dialogue from multiple knowledge sources. In Proceedings of the Human Language Technology Conference: 4th Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL), pages 201-208, Boston, MA. (This paper describes some of our prior work on automatically predicting student emotions in human-human tutoring.)

Koedinger: /* Independent variables */

2007-04-18T11:42:03Z

Independent variables

← Older revision		Revision as of 11:42, 18 April 2007
Line 68:		Line 68:
	=== Independent variables ===		=== Independent variables ===

	If a student answer is correct but uncertain, does the computer tutor respond with further dialogue (e.g., treat the answer as if it were incorrect) to reinforce the student’s understanding of the principle(s) under discussion? Note that in all conditions, incorrect answers (uncertain or not) are always responded to with additional sub-dialogue. This ~~approach~~ of adapting to uncertainty is similar to other kinds of [[error correction support]].		If a student answer is correct but uncertain, does the computer tutor respond with further dialogue (e.g., treat the answer as if it were incorrect) to reinforce the student’s understanding of the principle(s) under discussion? Note that in all conditions, incorrect answers (uncertain or not) are always responded to with additional sub-dialogue. This [[instructional method]] of adapting to uncertainty is similar to other kinds of [[error correction support]].

	The first control condition uses a non-adaptive version of the system, which only responds to incorrect student answers with further dialogue. The second control condition controls for the additional tutoring given in the experimental condition, by using a version of the system that responds to incorrect student answers with further dialogue and also responds to 15% of correct answers with further dialogue, to control for the additional tutoring in the experimental condition. This percentage represents the number of correct but uncertain student turns in our previously collected and annotated ITSPOKE corpora, as noted below.		The first control condition uses a non-adaptive version of the system, which only responds to incorrect student answers with further dialogue. The second control condition controls for the additional tutoring given in the experimental condition, by using a version of the system that responds to incorrect student answers with further dialogue and also responds to 15% of correct answers with further dialogue, to control for the additional tutoring in the experimental condition. This percentage represents the number of correct but uncertain student turns in our previously collected and annotated ITSPOKE corpora, as noted below.

Koedinger: /* Independent variables */

2007-04-18T11:36:47Z

Independent variables

← Older revision		Revision as of 11:36, 18 April 2007
Line 68:		Line 68:
	=== Independent variables ===		=== Independent variables ===

	If a student answer is correct but uncertain, does the computer tutor respond with further dialogue (e.g., treat the answer as if it were incorrect) to reinforce the student’s understanding of the principle(s) under discussion? Note that in all conditions, incorrect answers (uncertain or not) are always responded to with additional sub-dialogue.		If a student answer is correct but uncertain, does the computer tutor respond with further dialogue (e.g., treat the answer as if it were incorrect) to reinforce the student’s understanding of the principle(s) under discussion? Note that in all conditions, incorrect answers (uncertain or not) are always responded to with additional sub-dialogue. This approach of adapting to uncertainty is similar to other kinds of [[error correction support]].

	The first control condition uses a non-adaptive version of the system, which only responds to incorrect student answers with further dialogue. The second control condition controls for the additional tutoring given in the experimental condition, by using a version of the system that responds to incorrect student answers with further dialogue and also responds to 15% of correct answers with further dialogue, to control for the additional tutoring in the experimental condition. This percentage represents the number of correct but uncertain student turns in our previously collected and annotated ITSPOKE corpora, as noted below.		The first control condition uses a non-adaptive version of the system, which only responds to incorrect student answers with further dialogue. The second control condition controls for the additional tutoring given in the experimental condition, by using a version of the system that responds to incorrect student answers with further dialogue and also responds to 15% of correct answers with further dialogue, to control for the additional tutoring in the experimental condition. This percentage represents the number of correct but uncertain student turns in our previously collected and annotated ITSPOKE corpora, as noted below.

Koedinger at 11:29, 18 April 2007

2007-04-18T11:29:18Z

Show changes