Difference between revisions of "Note-Taking: Restriction and Selection"

From LearnLab
Jump to: navigation, search
(Further Information)
 
(37 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
=== Summary Table ===
 +
{| border="1" cellspacing="0" cellpadding="5" style="text-align: left;"
 +
| '''PIs''' || Aaron Bauer & Kenneth R. Koedinger
 +
|-
 +
| '''Other Contributers''' ||
 +
|-
 +
| '''Study Start Date''' || May 1, 2006
 +
|-
 +
| '''Study End Date''' || July 20, 2006
 +
|-
 +
| '''Laboratory Study''' ||
 +
|-
 +
| '''LearnLab Course''' || Causal and Statistical Reasoning (OLI)
 +
|-
 +
| '''Number of Students''' || 93
 +
|-
 +
| '''Total Participant Hours''' || 260
 +
|-
 +
| '''DataShop''' || Log data is not in DataShop. Should be available Fall 2007
 +
|}
 +
<br>
 +
 
== Abstract ==
 
== Abstract ==
We are conducting an investigation into the relationship between note-taking and learning in online courses. The literature has shown that the process of taking notes can have a positive impact on [[long-term retention]]. Our completed studies indicate that the [[features]] included in online note-taking applications can have an effect on these process benefits. Analyses of our results have led us to explore the effect of selection-based note-taking on both behavior and learning. We have the opportunity to develop note-taking applications that encourage active processing and retention.
 
  
We are exploring two hypothesis regarding the effect of note-taking on learning. First, we believe that note-taking encourages active processing, and thus long-term retention, when it requires students to attend to the critical elements of the learning material. This results in increased feature validity of the mental representation. Our second hypothesis is that note-taking facilitates long-term retention when it involves the creation of additional representations of concepts. ''These two hypotheses fit in with the fluency and coordinative learning clusters, respectively.'' Our studies are designed to evaluate these hypotheses by comparing note-taking tools offering different functionality.
+
This study evaluates two hypothesis regarding negative learning effects observed for note-taking tools with copy-paste functionality. Two novel tools are compared with two tools that use standard functionality: one that only allows copy-paste, and the other that only allows typing. The first novel tool restricts the amount of text students can select in any one copy-paste action, in order to increase attention to what is being recorded. The comparison between restricted and unrestricted copy-paste thus evaluates a [[feature focusing]] hypothesis within the [[Refinement and Fluency]] cluster. The second tool requires students to choose a reworded version of the idea selected, in order to evaluate whether it is important that students create their notes or simply view reworded versions of the ideas they want to record. The comparison between this tool and the other tools evaluates a [[Coordinative Learning]] hypothesis. Our study found that both novel tools performed worse on learning outcomes than the standard tools, which may be due to an observed user frustration with the novel interaction techniques. We also found that copy-paste only functionality increased the efficiency of learning relative to typing. In other words, people using the copy-paste tool learned the same amount in less time than users of the typing tool.
 +
 
 +
For information on the note-taking project see the [[Note-Taking_Technologies|top-level page.]]
  
 
== Glossary ==
 
== Glossary ==
Line 18: Line 41:
 
== Background/Significance ==
 
== Background/Significance ==
  
Note-taking research has shown that the process of taking notes can have a positive impact on long-term retention. Our preliminary studies have provided strong evidence that the features included in online note-taking applications impact how students take notes. We have also found that these differences can affect learning in several ways. These differences provide an opportunity to learn more about note-taking. Our main research question regards how and when note-taking increases long-term retention. By addressing this question, we have the opportunity to develop note-taking applications that encourage active processing and retention.  
+
In a previous study, we showed that copy-paste functionality in a note-taking application appears to reduce [[long-term retention]] relative to note-taking using typing or handwriting. In the previous study we found that when using copy-paste functionality, ideas that were recorded in a more wordy fashion were forgotten more than less wordy notes. Neither typed nor handwritten notes showed similar effects of wording. This study evaluates two hypotheses regarding these learning differences.  
  
There is extensive research on note-taking in both the educational psychology and educational technology fields. About half of note-taking studies have shown the process of note-taking to be beneficial to learning. Three-fourths of studies find that notes are a valuable resource when they can be reviewed (Kiewra, 1991). The cause of note-taking benefits is unclear, however. It has been attributed to rewording or summarizing, but experimental results are equivocal.
+
As the cost of wordy notes is low using copy-paste, we believed that wordy notes are indicative of decreased attention to the idea being recorded. In addition, while in typing and handwriting wordiness is associated with more time spent on the idea being recorded, this is not the case for copy-paste. We designed a tool to restrict the amount of text students could select in any one copy-paste action in order to increase the attention required to record a note. We believed this would result in superior learning. It is important to note that whereas in our previous study all tools had typing functionality, in this case only users of the typing tool could type.  
  
Note-taking technology provides new ways of studying note-taking, and provides investigators with more control over the note-taking process. This gives us the opportunity to study note-taking in further depth, perhaps elucidating the circumstances in which note-taking is beneficial. Unfortunately, few researchers have taken advantage of this opportunity. Most note-taking technology is developed to simply mimic paper-based practices, or provide new ways of taking notes.  
+
While students create their own notes through typing or handwriting, they simply copy material verbatim using copy-paste functionality. It may be that rewording of notes is important, or it may be that simply viewing an alternative representation of the learning material in notes is key to learning gains. To tease these apart, we designed a selection tool that, upon a student selecting an idea to record, gave the students three options to place in their notes. One was a reworded version of the idea recorded, and the other two were distractors. If it was only important to view an alternative wording, this tool should show superior performance to copy-paste. If involvement in the rewording was important, the tool should show equivalent performance to copy-paste.
  
These new devices have been shown to change the way students take notes, reinforcing the need to study how note-taking practices affect learning, if at all. Our investigations our a first step in doing so.
+
== Dependent Variables ==
 +
 
 +
'''Tests'''
 +
 
 +
[[Note:]] all tests include both multiple choice and free response questions. The multiple choice questions all involve solving problems (for example, given a response structure, which variables are direct causes of an effect, or which interact to produce an effect). In addition, some free response questions ask students to explain terminology used in the module.
 +
 
 +
''[[Normal post-test]]:'' Students are given a test immediately after studying the material. 
 +
 
 +
''[[Long-term retention]]:'' Students return a week following the treatment (which lasts between 30 and 90 minutes) to take this test.
 +
 
 +
''[[Long-term retention]] after review:'' After taking the long-term retention test, students are given their notes to review for 5 minutes. Following this review period, students take a final test.
  
== Dependent Variables ==
 
  
 
'''Behavior'''
 
'''Behavior'''
Line 40: Line 72:
 
''Experience:'' After taking the final test, students are given a survey which solicits their reaction to the tool they used. They are asked to identify their most and least favorite features of the tools, and how they believe the tool affected their note-taking behavior.
 
''Experience:'' After taking the final test, students are given a survey which solicits their reaction to the tool they used. They are asked to identify their most and least favorite features of the tools, and how they believe the tool affected their note-taking behavior.
  
'''Tests'''
+
== Independent Variables ==
 +
[[image:NT-TextEditor.gif]]
  
[[Note:]] all tests include both multiple choice and free response questions. The multiple choice questions all involve solving problems (for example, given a response structure, which variables are direct causes of an effect, or which interact to produce an effect). In addition, some free response questions ask students to explain terminology used in the module.
+
''This is the basic note-taking text-editor. Built in javascript, the editor occupies the lower third of the screen, while the learning materials occupy the top of the screen. The functionality of various manipulations is described below.  
 
 
''Normal Learning, immediate:'' Students are given a test immediately after studying the material.
 
 
 
''Long-Term Retention, Normal Learning:'' Students return a week following the treatment (which lasts between 30 and 90 minutes) to take this test.
 
 
 
''Normal Learning, review:'' After taking the long-term retention test, students are given their notes to review for 5 minutes. Following this review period, students take a final test.
 
 
 
== Independent Variables ==
 
  
 
'''Note-taking Treatment'''
 
'''Note-taking Treatment'''
Line 61: Line 86:
  
 
''Selection-Tool:'' Students can only create notes by choosing one of the 3 options made available when they select learning material.
 
''Selection-Tool:'' Students can only create notes by choosing one of the 3 options made available when they select learning material.
 +
<blockquote>'''Image of the Selection Tool'''<br>[[image:Note-SelectTool.gif|200]]</blockquote>
  
 
''Highlighting:'' Students can create notes by actively highlighting course material. These highlights are seen with differently colored backgrounds, as would be the case with a physical highlighter.  
 
''Highlighting:'' Students can create notes by actively highlighting course material. These highlights are seen with differently colored backgrounds, as would be the case with a physical highlighter.  
  
'''Student Variables'''
+
'''Context/Mediating Variables on Student Characteristics'''
  
 
''SAT Score:'' All students are asked to provide their SAT scores, as in previous studies SAT-Math was found to be an important covariate.  
 
''SAT Score:'' All students are asked to provide their SAT scores, as in previous studies SAT-Math was found to be an important covariate.  
Line 70: Line 96:
 
''Pretest score:'' Prior to the learning material, students take a pre-test similar to the normal tests described above.
 
''Pretest score:'' Prior to the learning material, students take a pre-test similar to the normal tests described above.
  
''Preferences:'' In the survey, students are asked how they prefer to take notes in their regular student-life.  
+
''Preferences:'' In the survey, students are asked how they prefer to take notes in their regular student-life.
  
 
== Hypotheses ==
 
== Hypotheses ==
 +
1. '[[Refinement]]': Restrictions on selections will increase the students' [[feature focusing|focus]] on the ideas being recorded, resulting in improved learning gains relative to unrestricted copy-paste.
  
1. ''Attention/Fluency-[[Refinement]]:''  Restricting the amount of material students can select in any individual selection behavior will increase the attention paid to critical features of the learning material. This will result in improved retention compared to unrestricted selection.
+
2. '[[Coordination]]': Viewing multiple versions of the same idea will require students to coordinate material, resulting in improved learning gains (less forgetting) relative to copy-paste.
 
 
2. ''Additional Representation/Coordinative Learning:'' Requiring students to create alternative representations of material will result in students having multiple representations of the learning material. This will result in improved retention when note-taking behavior does not create an alternate representation.
 
  
 
== Findings ==
 
== Findings ==
 +
[[Image:NT-Exp3-Learn.gif]]
  
- Copy-paste takes less time to produce the same learning results as Typing. Though copy-paste trends towards forgetting more at a delay than Typing, this effect was only significant in our first study.  
+
*''Learning:'' The two novel tools (bottom) showed significantly lower learning on all tests than did the standard tools(top two). The two novel tools (Selection and Restricted) were not significantly different from each other. The two standard tools (Copy-Paste and Typing) were not significantly different from each other). There was a significant overall effect for review (tests 2-3) across all conditions, but no significant effect for forgetting (tests 1 to 2).  
  
- Students who copy-paste do better when they take less wordy notes. This is not the case for other tools. This provides some evidence for the attention hypothesis.  
+
[[Image:NT-Exp3-Efficiency.gif]]
  
- Usability issues impaired our ability to directly address the attention and coordination hypotheses. Students disliked both the Selection and the Restricted-Paste tool, and used them less than the more standard Typing and Copy-Paste tools.  
+
*''Efficiency:'' The Paste condition showed more efficient learning than all other tools on the immediate test (1) and review test (3). Efficiency is defined as learning difference between two subsequent tests divided by the time spend studying the materials.
  
== Explanation ==
+
[[Image:NT-Exp3-Ideas.gif]]
  
The tools we designed were not effective in evaluating our hypothesis, as the results were somewhat confounded by student dissatisfaction with our novel tools. However, while there is little evidence that the selection tool will be effective, the negative impact of wordiness for unrestricted copy-paste may provide some indication that restricted copy-paste could be valuable if implemented properly. We are currently redesigning restrictions with two things in mind. First, we must create tools that students enjoy using. To do so, we will conduct usability and satisfaction studies on several restricted tools, in order to find the one most likely to be used. Secondly, we will design the restrictions themselves based on how effective copy-pasters self-restrict.  
+
*''Ideas Recorded:'' The Paste tool recorded significantly more total ideas than any other condition, and the same amount of key ideas as the typing condition. The Novel tools recorded significantly fewer key ideas than the standard conditions.
 +
*''Wordiness:'' In line with previous experiments, when students recorded wordier ideas using copy-paste functionality, they performed worse on test items associated with the idea than when they recorded less wordy ideas.
 +
*''Desirability:'' Students were significantly more dissatisfied with the selection tool than they were with any other tool. Most who used the restricted tool reported disliking the restrictions.
  
We will also conduct a simpler test of the additional representation hypothesis. All of the tools we have used so far have had students take notes in a separate window or sheet of paper. Therefore all get additional representations. In future studies we will test a highlighting condition, where students taking notes do not create a separate representation. This will allow us to separate the additional representation and attention hypotheses.
+
== Explanation ==
 +
*The results of this experiment do not support our hyptheses, as the novel tools performed worse on learning outcomes than did the standard tools. However, this may be due to usability issues. The Selection tool was the most disliked tool, and users reported disliking the restrictions in the restrited-paste tool. In addition, note-taking behavior was suboptimal, as the novel conditions recorded fewer key ideas than did the standard tools. It may be that the design of the novel tools required more attention to the process of note-taking, rather than the product.
 +
**Lesson: More attention must be paid to the design of note-taking interventions, in order to ensure adoption by users.
 +
*When students are only allowed to copy-paste, they appear to learn as much as when they type. Contrary to previous results, they do not forget more over time. In contrast to the previous experiment, students in the copy-paste condition could not type. This may have increased their attention to the process of pasting. In addition, students in the copy-paste condition showed reduced time on task compared to typing.
 +
**Lesson: Copy-Paste may be a more efficient note-taking technique than typing.
  
 
== Descendents ==
 
== Descendents ==
 +
[[Note-Taking_Technologies|Note-taking project page]]
  
None at this time
+
[[Note-Taking: Coordination]]
  
 
== Further Information ==
 
== Further Information ==
 
+
Bauer, A., Koedinger. K.R., Selection-Based Note-Taking Applications, ACM Symposium on Human Factors in Computing Systems 2007, accepted for publication [http://www.learnlab.org/uploads/mypslc/publications/bauer-chi2007.doc Word File, Final Version]
The following is a reference to a conference paper regarding our earlier study contrasting handwriting, typing, and copy-paste.
 
 
 
Bauer, A., Koedinger, K. Pasting and Encoding: Note-taking in Online Courses. IEEE International Conference on Advanced Learning Technologies (ICALT 2006), 5-7 July, Kerkrade, Netherlands.
 
 
 
- Bauer, A., Koedinger. K.R., Selection-Based Note-Taking Applications, ACM Symposium on Human Factors in Computing Systems 2007, in submission
 

Latest revision as of 18:21, 13 April 2007

Summary Table

PIs Aaron Bauer & Kenneth R. Koedinger
Other Contributers
Study Start Date May 1, 2006
Study End Date July 20, 2006
Laboratory Study
LearnLab Course Causal and Statistical Reasoning (OLI)
Number of Students 93
Total Participant Hours 260
DataShop Log data is not in DataShop. Should be available Fall 2007


Abstract

This study evaluates two hypothesis regarding negative learning effects observed for note-taking tools with copy-paste functionality. Two novel tools are compared with two tools that use standard functionality: one that only allows copy-paste, and the other that only allows typing. The first novel tool restricts the amount of text students can select in any one copy-paste action, in order to increase attention to what is being recorded. The comparison between restricted and unrestricted copy-paste thus evaluates a feature focusing hypothesis within the Refinement and Fluency cluster. The second tool requires students to choose a reworded version of the idea selected, in order to evaluate whether it is important that students create their notes or simply view reworded versions of the ideas they want to record. The comparison between this tool and the other tools evaluates a Coordinative Learning hypothesis. Our study found that both novel tools performed worse on learning outcomes than the standard tools, which may be due to an observed user frustration with the novel interaction techniques. We also found that copy-paste only functionality increased the efficiency of learning relative to typing. In other words, people using the copy-paste tool learned the same amount in less time than users of the typing tool.

For information on the note-taking project see the top-level page.

Glossary

Select/ion: This term is used in the context of this study to identify the behavior of using the mouse and cursor to actively highlight a portion of digital text. Selection is first step for several online note-taking techniques, including copy-paste and annotation.

Copy-Paste: This is the act of selecting material, copying it to the computer clipboard (via a keyboard shortcut or menu), and then pasting it into students’ notes.

Highlighting: The act of creating a lasting distinction between the selected text and the main content. For example, through creating a yellow background or underlining the text.

Research Question

Can we improve learning outcomes in online courses by changing the way students can take notes?

Background/Significance

In a previous study, we showed that copy-paste functionality in a note-taking application appears to reduce long-term retention relative to note-taking using typing or handwriting. In the previous study we found that when using copy-paste functionality, ideas that were recorded in a more wordy fashion were forgotten more than less wordy notes. Neither typed nor handwritten notes showed similar effects of wording. This study evaluates two hypotheses regarding these learning differences.

As the cost of wordy notes is low using copy-paste, we believed that wordy notes are indicative of decreased attention to the idea being recorded. In addition, while in typing and handwriting wordiness is associated with more time spent on the idea being recorded, this is not the case for copy-paste. We designed a tool to restrict the amount of text students could select in any one copy-paste action in order to increase the attention required to record a note. We believed this would result in superior learning. It is important to note that whereas in our previous study all tools had typing functionality, in this case only users of the typing tool could type.

While students create their own notes through typing or handwriting, they simply copy material verbatim using copy-paste functionality. It may be that rewording of notes is important, or it may be that simply viewing an alternative representation of the learning material in notes is key to learning gains. To tease these apart, we designed a selection tool that, upon a student selecting an idea to record, gave the students three options to place in their notes. One was a reworded version of the idea recorded, and the other two were distractors. If it was only important to view an alternative wording, this tool should show superior performance to copy-paste. If involvement in the rewording was important, the tool should show equivalent performance to copy-paste.

Dependent Variables

Tests

Note: all tests include both multiple choice and free response questions. The multiple choice questions all involve solving problems (for example, given a response structure, which variables are direct causes of an effect, or which interact to produce an effect). In addition, some free response questions ask students to explain terminology used in the module.

Normal post-test: Students are given a test immediately after studying the material.

Long-term retention: Students return a week following the treatment (which lasts between 30 and 90 minutes) to take this test.

Long-term retention after review: After taking the long-term retention test, students are given their notes to review for 5 minutes. Following this review period, students take a final test.


Behavior

Note-Quantity: The total number of ideas students place in their notes is captured, as well as the number of words used to express those ideas.

Note-Wording: How students word their notes is recorded. Each ideas is either recorded Verbatim, Abbreviated, or in students Own words.

Completion Time: The time students take to complete the learning material is recorded.

Motivation/Interest

Experience: After taking the final test, students are given a survey which solicits their reaction to the tool they used. They are asked to identify their most and least favorite features of the tools, and how they believe the tool affected their note-taking behavior.

Independent Variables

NT-TextEditor.gif

This is the basic note-taking text-editor. Built in javascript, the editor occupies the lower third of the screen, while the learning materials occupy the top of the screen. The functionality of various manipulations is described below.

Note-taking Treatment

Paste: Students can only create notes by copy-pasting material from the learning content to their notepad. Students can select as much material as they like in any single pasting action.

Restricted Paste: Students can only create notes by copy-pasting material from the learning content to their notepad. Students are restricted to selecting 90% of the words of any single sentence they are selecting.

Typing: Students can only create notes by typing directly into their notepad.

Selection-Tool: Students can only create notes by choosing one of the 3 options made available when they select learning material.

Image of the Selection Tool
200

Highlighting: Students can create notes by actively highlighting course material. These highlights are seen with differently colored backgrounds, as would be the case with a physical highlighter.

Context/Mediating Variables on Student Characteristics

SAT Score: All students are asked to provide their SAT scores, as in previous studies SAT-Math was found to be an important covariate.

Pretest score: Prior to the learning material, students take a pre-test similar to the normal tests described above.

Preferences: In the survey, students are asked how they prefer to take notes in their regular student-life.

Hypotheses

1. 'Refinement': Restrictions on selections will increase the students' focus on the ideas being recorded, resulting in improved learning gains relative to unrestricted copy-paste.

2. 'Coordination': Viewing multiple versions of the same idea will require students to coordinate material, resulting in improved learning gains (less forgetting) relative to copy-paste.

Findings

NT-Exp3-Learn.gif

  • Learning: The two novel tools (bottom) showed significantly lower learning on all tests than did the standard tools(top two). The two novel tools (Selection and Restricted) were not significantly different from each other. The two standard tools (Copy-Paste and Typing) were not significantly different from each other). There was a significant overall effect for review (tests 2-3) across all conditions, but no significant effect for forgetting (tests 1 to 2).

NT-Exp3-Efficiency.gif

  • Efficiency: The Paste condition showed more efficient learning than all other tools on the immediate test (1) and review test (3). Efficiency is defined as learning difference between two subsequent tests divided by the time spend studying the materials.

NT-Exp3-Ideas.gif

  • Ideas Recorded: The Paste tool recorded significantly more total ideas than any other condition, and the same amount of key ideas as the typing condition. The Novel tools recorded significantly fewer key ideas than the standard conditions.
  • Wordiness: In line with previous experiments, when students recorded wordier ideas using copy-paste functionality, they performed worse on test items associated with the idea than when they recorded less wordy ideas.
  • Desirability: Students were significantly more dissatisfied with the selection tool than they were with any other tool. Most who used the restricted tool reported disliking the restrictions.

Explanation

  • The results of this experiment do not support our hyptheses, as the novel tools performed worse on learning outcomes than did the standard tools. However, this may be due to usability issues. The Selection tool was the most disliked tool, and users reported disliking the restrictions in the restrited-paste tool. In addition, note-taking behavior was suboptimal, as the novel conditions recorded fewer key ideas than did the standard tools. It may be that the design of the novel tools required more attention to the process of note-taking, rather than the product.
    • Lesson: More attention must be paid to the design of note-taking interventions, in order to ensure adoption by users.
  • When students are only allowed to copy-paste, they appear to learn as much as when they type. Contrary to previous results, they do not forget more over time. In contrast to the previous experiment, students in the copy-paste condition could not type. This may have increased their attention to the process of pasting. In addition, students in the copy-paste condition showed reduced time on task compared to typing.
    • Lesson: Copy-Paste may be a more efficient note-taking technique than typing.

Descendents

Note-taking project page

Note-Taking: Coordination

Further Information

Bauer, A., Koedinger. K.R., Selection-Based Note-Taking Applications, ACM Symposium on Human Factors in Computing Systems 2007, accepted for publication Word File, Final Version