Amic Ho (2023). A Critical Overview of Studies on Eye Tracking and Visual Hierarchy. In: Cognitive Computing and Internet of Things. AHFE Open Access, vol. 73, pp. 179–187. https://doi.org/10.54941/ahfe1003295
This paper offers a thorough evaluation of recent eye-tracking studies in the area of visual hierarchy. The review’s specific goal is to investigate how eye-tracking technology might be used to study cognitive processes in the visual hierarchy while using pertinent characteristics. 30 papers, encompassing 36 investigations, were examined for this purpose. There are some noteworthy findings. Using eye-tracking technologies to investigate visual hierarchies has gained a lot of attention recently. Typically, the research included university students, scientific content, and chronological frequency count ranges of eye-tracking metrics. Insights into cognitive processes including selection, organisation, and integration might be gleaned by monitoring eye movement patterns. Visual hierarchy principles, visual content, individual differences; and visual hierarchy principles, visual content. To provide recommendations for future research and practises, specific gaps in the literature and implications of previous results on visual hierarchy design were also identified.
Diverse hierarchy environments use visual hierarchy elements, including visual and vocal instructional material. Visualisation technologies allow edu- cators to use static or dynamic graphics in face-to-face classrooms (Mayer, 2014). Mobile technology and online courses allow learners to access visual resources anytime, anywhere. Despite their widespread usage, visual hiera- rchy cognitive exercises have only recently been noticed (Alemdag & Cagiltay, 2018). Interviews, behavioural evaluations, and self-reporting have generally been used to infer various approaches to hierarchy information processing (Rodrigues & Rosa, 2019). Self-reporting cannot capture cognitive pro- cess temporal variations. A visual hierarchy requires cognitive processing metrics (Moreno, 2006). Eye tracking may be employed to study cogni- tive activity in the visual hierarchy without self-reporting. This tool can test hypotheses about where people gaze during text–picture integration (Tabbers et al., 2008). Based on learners’ information-processing capacities, instructors may improve visual products using eye-tracking data. However, the information that interprets eye-tracking measures as learners’ cogni- tive activity is not adequate (Lai et al., 2013). In their literature review, eye-tracking technology in hierarchy research was discussed to address this
gap. The authors categorised eye tracking research by cognitive develo- pment and hierarchy concerns. Although most research has focused on how different teaching methods affect students’ conceptual, perceptual, and lin- guistic growth, students’ eye-tracking data reveals both the approaches they use to analyse information and the results of those strategies (Lai et al., 2013). They explored general hierarchy subjects using eye-tracking tech- nology; however, they could not explain visual-hierarchy cognitive proces- ses. Hence, visual hierarchy research must examine information processing and factors that impact knowledge acquisition or creation (van Gog & Jarodzka, 2013). To gather and combine the latest results on this issue, it is essential to review research published in multiple databases after 2006. This paper evaluates recent studies on eye-tracking technologies in the visual hierarchy in many databases. It describes how eye-tracking measu- res are used in visual hierarchy research, what factors may affect those measures, and how eye-tracking measures relate to the effectiveness of a hierarchy.
USING BULLET POINTS IN TYPOGRAPHY The components of visuals are text and images (Mayer, 2014). Words may be spoken or written. Diagrams, pictures, and drawings make up static graphics, while animation and video make up dynamic graphics. Construction of men- tal representations from text and visuals is a component of visual hierarchy. Students may learn by hearing and seeing through visual presentations, e- hierarchy, computer games, simulations, and virtual-reality settings (van Gog & Jarodzka, 2013). According to research on visual hierarchy, instructional messages containing visuals that are consistent with the human brain system may enhance the hierarchy. How individuals gain information using instructi- onal visuals is described by the cognitive theory of hierarchy (Mayer, 2014). This theory assumes limited-capacity active dual-channel processing. Before anything else, there is the dual-channel idea, which states that people’s visual and auditory systems operate independently. Second, according to ideas like Sweller’s cognitive load theory and Baddeley’s working-memory model, there is a limit to the amount of knowledge a person can receive via each channel at once. Last but not least, the active-processing assumption claims that peo- ple deliberately choose, arrange, and integrate data to create coherent mental representations (Chuang & Liu, 2012). No concrete proof of the cognitive processes involved in the visual hierarchy exists, despite the fact that the cognitive theory of visual hierarchy has produced a number of fundamental and sophisticated visual design principles (Duchowski, 2007). Measurement of cognitive processes, which include visual attention, cognitive load, and visual search, is challenging (Park et al., 2015). Hence, eye tracking may help in the study of attention allocation during visual hierarchy processes’ or simi- lar. Eye-tracking technology allows researchers and practitioners to measure the visual attention of students to analyse multimodal information processing (Frankel et al., 2012).
A Critical Overview of Studies on Eye Tracking and Visual Hierarchy 181
EYE-TRACKING TECHNOLOGY IN VISUAL HIERARCHY RESEARCH Eye tracking may reveal a person’s interests and perception of a scenario (Duchowski, 2007). Eye tracking detects fixations and saccades. Fixations reflect eye stability. Eye fixations indicate focus, according to the eye–mind theory (Rayner, 1998). Saccades, as rapid eye movements between fixations, show visual attention changes (Azevedo & Aleven, 2016). Lai et al. clas- sify that fixations vary from saccades chronologically, geographically, and numerically. A temporal scale measures eye movement. Total, average, and initial fixation durations are eye-movement measurements. Spatial scale mea- surement, locations, distances, orientations, sequences, transactions, spatial organisation, and interactions between fixations or saccades are the factors potentially influence the flow of the eye movement. Saccade length and fixa- tion order measure spatial size. The counting scale measures eye movement. Fixation and inter-scanning-count scales assess eye movement. Fixations, saccades, pupil size, and blink rate are also measured. Because of the conne- ction between eye movements and cognitive processes, these eye-movement metrics may be utilised to investigate the many ways in which students learn (Frankel et al., 2012). Eye-movement measures may indicate visual atten- tion on scene components or shifts in visual-attention focus (Frankel et al., 2012), depth of information processing (Park et al., 2015), and information processing difficulties. Education academics are just now employing eye- tracking equipment to uncover cognitive processes. The visual hierarchy of eye movements has been studied extensively (Chuang & Liu, 2012). Hence, visual-hierarchy cognitive processes must be studied using eye tracking. This will validate various approaches to visual hierarchy research. Eye tracking research may uncover literature gaps and visual-hierarchy environment ideas. This article synthesises multimodal hierarchy eye tracking research. These are the main and secondary research questions that guide this study:
How are cognitive processes associated with eye-tracking measures in visual hierarchy studies?
What affects eye-tracking metrics in visual hierarchy studies? And in visual hierarchy studies, how does eye tracking affect success?
RESEARCH METHOD AND PROCESS Scholars everywhere utilise databases such as Web of Science, Education Resources Information Center, Education Source, and PsycINFO to access primary sources, so the same was done for this study. The terms ‘visual’, ‘hie- rarchy’, ‘eye movement’, ‘eye tracking’, ‘gaze movement’, and ‘gaze tracking’ were all merged in this literature search. To discover the most relevant and current research, a literature search was conducted using the aforementio- ned databases, which focused on articles published in peer-reviewed journals between 2006 and 2021. Articles published in English with no translations were excluded from this review. First, 156 items were found and 76 dupli- cate articles were removed from several databases. Second, the remaining 80 items were checked for inclusion. This review included unique visual hie- rarchy research that employed eye-tracking technologies. Students also had to
182 Ho
answer two literature-review sub-research questions. The titles and abstracts of 17 publications violated the inclusion criteria and were excluded. Lastly, the remaining 53 papers’ entire texts were critically assessed to verify they answered this researcher’s concerns. The review removed 13 articles: seven did not employ the technology of eye tracking within the cognitive processes, two did not study visual system usability, and two did not use visual-hierarchy settings. This review included 30 publications after the study selection and included 36 studies, since several journals reported multiple experiments or investigations.
FINDINGS AND DISCUSSION The cognitive theory of visual hierarchy states that choosing, organising, and integrating form coherent mental representations (Mayer, 2014). This theory can examine visual hierarchy studies’ eye movements (Glaser & Schwan, 2015). Three of the visual-hierarchy cognitive processes studied used eye-movement measures. This study focused on the processing of on- screen phrases and visuals called areas of interest (AOIs) using eye-tracking technologies.
SELECTION Students choose relevant visuals to keep in their working memory. Students’ early fixations on visual objects might be utilised to enhance visual search and salience (Schueler et al., 2011). Several of the evaluated research studies assessed learners’ use of visual search throughout the selection process by timing how long it took them to fixate on a relevant word or picture (Schueler et al., 2011) or how rapidly they focused on a cued object (Moreno, 2006). Also measured was the percentage of fixations on diagrams or text, the time spent on the first pass on text or pictures, and the location of the first five fixations (Mason et al., 2015) to gain a sense of how differentially attention is spread between both words and pictures throughout initial processing.
ORGANISATION Students create working memory models using words or visuals. Fixing pro- cessed information requires organisation (Rayner, 1998). Textual or pictorial AIO fixation length and number may indicate attention. Rayner (1998) implies deeper processing with a longer focus. Fixation time predicted proces- sing depth in many studies. Counting AOI fixations helps researchers evaluate learners’ processing intensity. Fixation measurements and scan pathways identified learners’ attention sequencing when organising words or pictures. Fixation time shows cognitive effort and processing depth. Longer mean sti- mulus fixation periods may indicate processing issues. Yang et al. (2013) inferred processing obstacles from mean fixation time. So, cognitive pro- cessing of fixation duration differs. Pupil dilation also indicates cognitive strain. Only Chuang and Liu (2012) estimated cognitive burden in segmen- ted visual and assessment pages using pupil size. Pupil size is sensitive to light and brightness; thus researchers should be careful.
A Critical Overview of Studies on Eye Tracking and Visual Hierarchy 183
INTEGRATION Students integrate by connecting graphical and verbal models to relevant historical knowledge (Mayer, 2014). Students’ text-to-picture transitions may show integration (Takacs & Bus, 2016). In the study, text-to-pictorial switch- ing exhibited integration. Text and images were split among numerous AOIs to achieve visual integration (Moreno, 2006). In a certain study, transitions between two text segments or two pictures were considered integrating pro- cesses. Lastly, scan pathways were subjectively analysed to reveal learners’ integration processes plus textual and graphical reading habits. Integrated transitions showed visual integration issues. Fewer transitions between two distant images increased the working memory load. These transitions linked information overload to video–text switching (Wang et al., 2016).
POTENTIAL FACTORS THAT CAN AFFECT EYE-TRACKING MEASUREMENTS IN VISUAL HIERARCHY Visual hierarchy principles, visual material, individual differences, metaco- gnition, and emotions may alter eye-tracking measures (Figure 1). Emotions were less studied than visual hierarchy concepts. These papers examined their potential factors.
Figure 1: Potential factors that can affect eye-tracking measurements in visual hierarchy.
184 Ho
Visual Hierarchy Principles Visual hierarchy research may be supported by eye-movement data. Most of the analysed research used eye tracking to explore how visual hiera- rchy principles affect cognitive functions. Researchers have focused on the signalling or cueing principle, then comes physical proximity, then cohe- rence, and finally attractiveness according to research on visual hierarchy (n = 9). Students’ ability to concentrate increased when they began using signals. According to the signalling or cueing principle, students perform bet- ter on tests when signals draw their attention to important information or highlight the sequence in which it is presented. Using this method, Mason et al. (2015) discovered that those exposed to eye movement modelling exam- ples had more integrative text-to-picture transitions than those exposed to a control condition. Analysed studies showed that the modality principle improved visual attention. According to the idea of modalities, combining verbal descriptions with visual ones might help establish a clearer chain of command (Schueler et al., 2011). This analysis looked at how students per- form in visual-hierarchy settings that include either oral or written material. The most frequent result was that students learning from spoken text found processing visuals more time-consuming than students learning from writ- ten text. Attractive details and spatially separated visual components often detracted from the integration of important visual elements.
Visual Content Visual content – the type of visual element, graphics and text – also affected eye-tracking measures. Nine publications examined attentional emphasis on text and pictures by visual-element type (Azevedo & Aleven, 2016). Studies found that students focus more on words than graphics. This suggests visual learners prefer text-based hierarchies. Two research studies evaluated the impact of visual and spatial texts. Due to the visuospatial sketchpad’s limi- ted capacity, digesting text with spatial information and synchronising eye movements might hinder hierarchy. The scientists proposed that that kind of text content does not affect processing difficulties, since they discovered no significant influence on fixation time. The studies compared the effects of 2D and 3D visuals (Glaser & Schwan, 2015), static and dynamic visuals, photos and conceptual graphics (Moreno, 2006), and content and context visuals (Chen et al., 2015). Learners focused more on dynamic than static graph- ics. Chen et al. (2015) discovered that learners spent more time analysing three-dimensional dynamic atomic orbital visualisations and made more fixa- tions. Takacs and Bus (2016) found that kindergarten children paid greater attention to motion visuals than static drawings in a tale.
Individual Differences Since there were many forms of individual characteristics to explore, they were treated under a new subject. Of the evaluated publications, 13 addressed how individual variations affect information processing. Most also examined how individual variations affect other parameters, such as prior knowledge (Lai et al., 2013), spatial ability, attentional control/level (Mason et al., 2015),
A Critical Overview of Studies on Eye Tracking and Visual Hierarchy 185
and working memory capacity. Prior knowledge differentiated various appro- aches to cognitive processing. Those with greater experience spent more time reading the text and looking at the visuals during the first pass, but during the second pass they paid more attention to the photographs that were related to the topic. According to Yang et al. (2013), students with domain expertise paid greater attention to texts, keywords, and certain images, and they inte- grated text and pictures more. The results on spatial ability and information selection and processing were inconsistent with existing understanding.
Metacognition Metacognition – including monitoring, control, and strategy design – also affects visual hierarchy eye movements. Metacognitive assistance helps stu- dents analyse, process, and assess hierarchy in interactive visual contexts. Metacognitive variables regulate cognitive processes and emotions to medi- ate hierarchy (Moreno, 2008). Researchers have examined visual hierarchy metacognition. Metacognition was seldom mentioned using eye-tracking technologies in the examined papers. Metacognition was found to affect visual attention in four studies. Techniques for metacognition were developed for use in situations with visual hierarchies, and they included self-monitoring questions and assistance for cognitive hierarchy. The use of hierarchy is help- ful for dynamic presentation styles, since it increases the amount of material that can support transitions. Alternatively, implementation goals or if-then strategies can help students understand visual education. In the visual hie- rarchy, text-picture integration and diverse implementation goals increased picture-text transitions. Metacognitive monitoring and control from frequent studying and testing and image requests affected visual attention allocation.
Emotions In the cognitive-affective theory of visual hierarchy, mediation by emoti- ons enhances or decreases cognitive engagement (Scheiter & Eitel, 2018). Positivity is either an ‘extraneous cognitive load’ or a ‘hierarchy facilitator’. Just three research studies looked at how inducing emotions or designing for emotions affected eye-tracking data. Emotions may either stimulate or retard thought processes in the visual hierarchy. In the analysed experiments, emotions were induced using either pre-visual-hierarchy emotional-induction strategies (such as listening to music or recalling a happy/sad experience) or in-environment emotional-design characteristics (such as anthropomorphi- sms and educational agents with emotional expressions). Positive emotion induction was related to decreased hours spent and less concentration on important information than negative emotion induction. This controversy calls for further study.
CONCLUSION
This study synthesises eye tracking studies on the visual hierarchy to demon- strate how it might be applied. The study identifies growing interest in eye-tracking technologies in visual hierarchy studies. These investigations
186 Ho
used college students, scientific texts, and temporal and count scale eye- movement measures. The examined research employed eye tracking to infer cognitive processes of choosing, organising, and integrating. Visual-hierarchy principles, content, individual differences, metacognition, and emotions might affect eye-movement measures. It was found that metacognition and emotion were little studied. Finally, eye tracking assessments of cognitive processes and hierarchy performance were linked. Eye tracking studies may help students study using visual content. This article explains how eye- tracking technology has helped researchers explore visual processing with relevant characteristics. This study may inform visual hierarchy, researchers, and practitioners. The study proves the application of eye-tracking techno- logy in educational research to analyse hierarchical processes as opposed to outcomes.
ACKNOWLEDGMENT
The work described in this paper was fully/substantially/partially suppor- ted by Hong Kong Metropolitan University Research Grant (No. HE- RD/2022/1.21).
REFERENCES
Alemdag, E., & Cagiltay, K. (2018). A systematic review of eye tracking research on multimedia learning. Computers & Education, 125, 413–428. Azevedo, R., & Aleven, V. (Eds.). (2016). International handbook of metacognition
and learning technologies. New York, NY: Springer. Chen, S. C., Hsiao, M. S., & She, H. C. (2015). The effects of static versus dynamic 3D representations on 10th grade students’ atomic orbital mental model construction: Evidence from eye movement behaviors. Computers in Human Behavior, 53, 169–180. doi: 10.1016/j.chb.2015.07.003 Chuang, H. H., & Liu, H. C. (2012). Effects of different multimedia presen-
tations on viewers’ information-processing activities measured by eye-tracking technology. Journal of Science Education and Technology, 21(2), 276–286. doi: 10.1007/s10956-011-9316-1 Duchowski, A. (2007). Eye-tracking methodology: Theory and practice (2nd ed.).
London, England: Springer. Frankel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education. New York: The McGraw-Hill Companies, Inc. Glaser, M., & Schwan, S. (2015). Explaining pictures: How verbal cues influe-
nce processing of pictorial learning material. Journal of Educational Psychology, 107(4), 1006–1018. doi: 10.1037/edu0000044 Lai, M. L., Tsai, M. J., Yang, F. Y., Hsu, C. Y., Liu, T. C., Lee, S. W. Y., Tsai, C. C. (2013). A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review, 10, 90–115. doi: 10.1016/j.edurev.2013.10.001 Liu, H. C., & Chuang, H. H. (2011). An examination of cognitive processing of
multimedia information based on viewers’ eye movements. Interactive Learning Environments, 19(5), 503–517. doi: 10.1080/10494820903520123
A Critical Overview of Studies on Eye Tracking and Visual Hierarchy 187
Mason, L., Pluchino, P., & Tornatora, M. C. (2015). Eye-movement modeling of integrative reading of an illustrated text: Effects on proces- sing and learning. Contemporary Educational Psychology, 41, 172–187. doi: 10.1016/j.cedpsych.2015.01.004 Mayer, R. E. (2014). The Cambridge handbook of multimedia learning (R. E. Mayer, Ed.). New York, NY: Cambridge University Press. Mayer, R. E. (2017). Using multimedia for e-learning. Journal of Computer Assisted
Learning, 33(5), 403–423. doi: 10.1111/jcal.12197 Moreno, R. (2006). Does the modality principle hold for different media? A test of the method-affects-learning hypothesis. Journal of Computer Assisted Learning, 22(3), 149–158. doi: 10.1111/j.1365–2729.2006.00170.x Park, B., Knörzer, L., Plass, J. L., & Brünken, R. (2015). Emotional design and positive emotions in multimedia learning: An eyetracking study on the use of anthropomorphisms. Computers & Education, 86, 30–42. doi: 10.1016/j.compedu.2015.02.016 Rayner, K. (1998). Eye movements in reading and information processing: 20 years
of research. Psychological Bulletin, 124(3), 372. Rodrigues, P., & Rosa, P. J. (2019). Eye-tracking as a research methodology in educational context. T. M. Bloggs (Ed.), In Early childhood development (pp. 269–294). doi: 10.4018/978-1-5225-7507-8.ch013 Schüler, A., Scheiter, K., & Gerjets, P. (2011). Does the modality effect in multime-
dia learning appear only with text containing spatial information? Zeitschrift für Pädagogische Psychologie, 25(4), 257–267. doi: 10.1024/1010-0652/a000050 Scheiter, K., & Eitel, A. (2018). The use of eye-tracking as a research and instructional tool in multimedia learning. In Digital multimedia (pp. 698–719). doi: 10.4018/978-1-5225-3822-6.ch035 Tabbers, H. K., Paas, F., Lankford, C., Martens, R. L., & van Merriënboer, J J. G.
(2008). Studying eye movements in multimedia learning. In Understanding multimedia documents (pp. 169–184). doi: 10.1007/978-0-387-73337-1_9 Takacs, Z. K., & Bus, A. G. (2016). Benefits of motion in animated storybooks for children’s visual attention and story comprehension: An eye-tracking study. Frontiers in Psychology, 7. doi: 10.3389/fpsyg.2016.01591 van Gog, T., & Jarodzka, H. (2013). Eye-tracking as a tool to study and enha-
nce cognitive and metacognitive processes in computer-based learning environ- ments. In International handbook of metacognition and learning technologies (pp. 143–156). doi: 10.1007/978-1-4419-5546-3_10 Yang, F. Y., Chang, C. Y., Chien, W. R., Chien, Y. T., & Tseng, Y. H. (2013). Tracking learners’ visual attention during a multimedia presentation in a real classroom. Computers & Education, 62, 208–220. doi: 10.1016/j.compedu.2012.10.009
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Cognitive Computing and Internet of Things
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-49-0
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional