Visual-verbal integration

Brief statement of principle

Visual-verbal integration principle: Instruction that includes both visual and verbal information leads to robust learning (the development of coherent, flexible knowledge representations) when the instruction supports learners as they coordinate information from both sources and the representations guide student attention to deep features.

Description of principle

Operational definition

Instruction encourages students to link or coordinate visual information (e.g., diagrams) and verbal information (e.g., text) by:

• Supporting direct interaction with diagrams during problem solving
  • For more information, see Butcher & Aleven studies: Contiguous Representations; Elaborated Explanations; Integrated Hints
• Presenting diagrams that make explicit key features of an expert mental model
  • For more information, see Davenport et al. studies

Examples

In geometry, students need to connect the conceptual definition of a geometry principle (e.g., a verbal description of "Vertical Angles") with the relevant visual diagram features and configurations (e.g., the visual instantiation of "Vertical Angles" formed by two crossing lines where the angles share a common vertex but no common sides). Visual-verbal integration can be tested by having students analyze the appropriateness of geometry rules to a particular diagram.

Mayer (2001) has other examples of the related "multimedia principle".

Experimental support

Laboratory experiment support

Prior research has shown that students benefit from activities that coordinate both visual and verbal sources; these activities include verbal comparison of self-generated and ideal diagrams (Van Meter, 2001; Van Meter, Aleksic, Schwartz, & Garner, 2006) as well as dragging and dropping verbal information into a diagram to create an integrated representation (Bodemer, Ploetzner, Feuerlein, & Spada, 2004).

Even relatively simple forms of coordination between visual and verbal information can impact student learning. Benefits have been found for the temporal association of visual and verbal information, where presenting visual and verbal information at the same time leads to better learning than presenting them at different times (Mayer & Anderson, 1992; Mayer, Moreno, Boire, & Vagge, 1999). Research also has identified the importance of spatial association, where learning is supported by placing visual and verbal materials in close physical proximity or integrating them into a single, combined representation (Hegarty & Just, 1993; Mayer, 1989; Moreno & Mayer, 1999).

In vivo experiment support

Butcher and Aleven's (2007; submitted) in vivo research has demonstrated that the addition of interactive diagrams to an intelligent tutor in geometry supports deep understanding of geometry principles and long-term retention of problem-solving skills. The interactive diagrams were designed as a method to support visual-verbal integration; they allow students to work directly with the diagrams during problem solving. Results show that students who used the interactive diagrams are better able to work with new diagrams and geometry principles to 1) explain when and why geometry principles are inappropriately applied to a diagram, and 2) to explain how unsolvable problems could be made solvable. For more details on these studies, please see Contiguous Representations for Robust Learning (Aleven & Butcher) and Using Elaborated Explanations to Support Geometry Learning (Aleven & Butcher).

Butcher and Aleven also have been studying scaffolds that directly connect relevant visual and verbal information. Results of these studies are ongoing; for more information, please see Mapping Visual and Verbal Information: Integrated Hints in Geometry (Aleven & Butcher) and Visual Feature Focus in Geometry: Instructional Support for Visual Coordination During Learning (Butcher & Aleven).

Davenport et al. (2007, 2008) tested the role of visual-verbal integrate in chemistry instruction and found that instruction that includes diagrams and text only leads to learning gains when the representations are clearly aligned with an expert mental model. A knowledge decomposition of chemical equilibrium (informed by Lab Studies #1 and #2) as well as discussions with our Chemistry working group revealed that a key knowledge component of "progress of reaction" is left implicit in many types of traditional chemistry instruction. In one study two sets of online lectures were created by Prof. Yaron to determine if instruction that uses multiple representations to convey the notion of progress of reaction would lead to more robust learning of chemistry concepts. Traditional instruction described equilibrium using chemical notations and text, the New instruction described equilibrium using molecular diagrams depicting the progress of reaction. Transfer measures of open-ended responses and conceptual multiple choice questions were collected and revealed that diagrams that were aligned with the progress of reaction framework increased learning, particularly for low knowledge students. For more information about additional studies see: Visual Representations in Science Learning, Davenport, Klahr & Koedinger.

Theoretical rationale

One proposed theoretical rationale for visual-verbal coordination benefits is that temporal and spatial coordination reduces the cognitive load demands associated with working memory maintenance and visual search (Mayer, 2001). The reduction in cognitive effort needed to find and maintain multiple sources of information allows students to engage in deeper processing.

However, reducing cognitive load in and of itself does not mean that students will engage in robust learning processes. Another interpretation of the learning benefits found when materials support connections between visual and verbal representations is that these materials prompt students to engage in cognitive processing that integrates visual and verbal information with existing knowledge representations. That is, support for visual-verbal integration prompts student to engage in active processes that support deep understanding, such as self-explanation or other sense-making processes. Previous research has found that adding diagrams to a text increases the number of correct inferences that integrate to-be-learned information (Butcher, 2006).

Conditions of application

Instruction that promotes Visual-verbal integration will only be successful if students actively process information from both the pictures and text and if the informational content of pictures and text are clearly aligned with instructional objectives.

• Visual Representations Must Target Deep Features
  • In both in vivo and labs studies, Davenport et al. wiki page found that pictures that were not aligned to an expert model of equilibrium processes did not support learning beyond text alone.
  • [Others have reported not replicating Mayer's multi-media principles, like Jennifer Wiley -- we should track down the reference]

• Visual-Verbal Information Should be Actively Integrated
  • Butcher & Aleven wiki page found that adding explanations that linked geometry principles to diagram features did not improve learning beyond direct interaction with diagrams. Log data analysis suggests that the visually-related explanations may not have been actively processed, especially when students were already working with the diagrams.

• Format is Less Important than Content
  • Visual representations in a variety of formats can support learning, as long as the informational content is relevant and consistent Static vs. Animated Visual Representations for Science Learning (Kaye, Small, Butcher, & Chi)

Variations (descendants)

The multimedia principle in Mayer (2001) is a variation or a kind of visual-verbal integration.

Generalizations (ascendants)

This principle is within the Coordinative Learning cluster.

References

Bodemer, D., Ploetzner, R., Feuerlein, I., & Spada, H. (2004). The active integration of information during learning with dynamic and interactive visualisations. Learning and Instruction, 14, 325-341.

Butcher, K. R. (2006). Learning from text with diagrams: Promoting mental model development and inference generation. Journal of Educational Psychology, 98, 182-197.

Butcher, K., & Aleven, V. (2007). Integrating visual and verbal knowledge during classroom learning with computer tutors. In D.S. McNamara & J.G. Trafton (Eds.), Proceedings of the 29th Annual Cognitive Science Society (pp. 137-142). Austin, TX: Cognitive Science Society. PDF File

Butcher, K., & Aleven, V. (submitted). Diagram Interaction during Intelligent Tutoring in Geometry: Support for Knowledge Retention and Deep Transfer. Submitted to CogSci 2008. Link to PDF

Davenport, J. L., Yaron, D., Klahr, D., & Koedinger, K. (2008). When do diagrams enhance learning? A framework for designing relevant representations. Paper accepted for the 2008 International Conference of the Learning Sciences, June 2008. download

Davenport, J.L., Klahr, D. & Koedinger (2007). The influence of diagrams on chemistry learning. Paper presented at the 12th Biennial Conference of the European Association for Research on Learning and Instruction. August 2007. download

Hegarty, M. & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32, 717-742.

Mayer, R. E. (1989). Systematic thinking fostered by illustrations in scientific text. Journal of Educational Psychology, 81, 240-246.

Mayer, R. E. (2001). Multimedia learning. Cambridge: Cambridge University Press.

Mayer, R. E. & Anderson, R. B. (1992). The instructive animation: Helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology, 84, 444-452.

Mayer, R. E., Moreno, R., Boire, M. & Vagge, S. (1999). Maximizing constructivist learning from multimedia communications by minimizing cognitive load. Journal of Educational Psychology, 91, 638-643.

Moreno, R. & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91, 358-368.

Van Meter, P. (2001). Drawing construction as a strategy for learning from text. Journal of Educational Psychology, 93(1), 129-140.

Van Meter, P., Aleksic, M., Schwartz, A., & Garner, J. (2006). Learner-generated drawing as a strategy for learning from content area text. Contemporary Educational Psychology, 31, 142-166.

See also integration and coordination.