Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Cell Phone Conversations with Hands-Free Devices Interfering with Cognition of Visual Information while Driving

Graduate School of Information and Telecommunication Engineering, Tokai University, Takanawa, Tokyo 108-0074, JAPAN ↗
Publication year
Digital Object Identifier10.54941/10046

Abstract

Hands-free devices do not shift the line of vision from the road to a cell phone and limit the person driving. So, the use of hands-free devices while driving is permitted by law. However, it is conceivable that even the interaction with auditory information interferes with cognition of the visual information because the limited capacities of attention resources. Thus, this research discusses whether auditory information interactions interfere with cognition of visual information. In this research, the dual task method was carried out. This research obtained the reaction times and the frequency of oversight of signals from the primary task and the information by means of the NASA-TLX as the subjective evaluations. These data indicated that delays in reaction to signals, increases in oversight of signals, and so on were caused by the secondary task. Thus, it was suggested that cognition of visual information was disturbed by the auditory information interaction. Therefore, the use of a cell phone with a hands-free device interferes with driving.

INTRODUCTION

Recently, portable terminals such as smart phones are owned by many people because communication technology is developing. Smart phones are superior in portability, can telephone, and can send e-mails. Moreover, one characteristic is that the user is able to add many functions which they believe are essential to smart phones, such as car -navigation, and a map. Because of this smart phones are useful at various places and in various situations because they are more convenient. However, they are prohibited in certain places and situations. For instance, the use of a smart phone or a cell phone while driving a car is prohibited. Today, it is prohibited by law to use a cell phone while driving a car in advanced countries. The line of vision being needed to shift from the road to a cell phone is a situation called "inattentive driving". Furthermore, the persons are limited by using a cell phone. Inattentive driving is especially hazardous for driving, because drivers make various decisions based on the visual information, and the lack of the visual information may lead to serious accidents. Recently, cell phones have enabled us to use hands-free devices. Hands-free devices only need the auditory information interaction. Thus, the operations do not have to shift the line of vision from the road to a cell phone and limit the driver to operate a cell phone and make a call. So, the use of hands-free devices while driving a car is safe and permitted by law in advanced countries. However, the persons can only pay attention to limited information because they have attention resources which have limited capacities (Akira and Satoru, 2001). If a person pays attention to the auditory information interaction, it may hinder the processing of the visual information by not paying enough attention to the visual information. It may cause cognition and reaction of the visual information to be delayed, even if a shifting of line of vision from the road to the cell phone is not needed. Therefore, it is thought that the conversation itself interferes with the cognition of visual information which is the central work of driving. Even the use of a cell phone with a hands-free device while driving a car may cause delays in reaction to signals and oversight of signals. However, in- person conversations do not cause delays, because in-person conversations are adjusted by the driving condition (Parks, 1991). Therefore, this research focused on the interference of the use of a cell phone with a hands- free device with driving a car when the driving conditions are not understood, and was carried out by the dual task method. In this study, the purpose was to investigate the interference of the auditory information interaction to the cognition of the visual information by the dual task method.

THE PROPERTIES OF HUMAN COGNITIVE PROCESSING Working Memory

The working memory has the most central role in human cognitive processing. Various models of the working memory are suggested, and the most famous model in them is the working memory model by Baddeley. This model is composed by four subsystems which are the central executive, the visuospatial sketchpad, the phonological loop, and the episodic buffer (Alan, 2000). They are shown in Figure 1.

Figure 1. Working Memory Model
Original PDF Figure 1.

The phonological loop is composed by the phonological store and the articulatory rehearsal component. The phonological store is able to retain the acoustic or the phonological code which is presented by the auditory information only for a temporary amount of time. The acoustic or the phonological code, which is input to the phonological store, is forgotten when it passes the temporary time unless the acoustic or the phonological code is rehearsed. The articulatory rehearsal component is composed by overt or covert vocalization, and has the function which the auditory information is maintained by sub-vocal rehearsal. The visuospatial sketchpad is composed by the visuospatial store and the rehearsal component like the phonological loop. The visuospatial store is able to retain the code of visual space. However, there are two stores in the visuospatial sketchpad to maintain the characteristic of the visual object and the information about the spatial location (Alan, 2012). Moreover, the function which is to integrate various types of information was named the episodic buffer by Baddeley. It is thought that the episodic buffer enables the interaction of the information which is coded by various dimensions, and is a subsystem to enable integration of the working memory and the long-term memory. For instance, when an object as the visual information is shown and a person answers about it by vocalization, the information which is shown is input into the visuospatial sketchpad. Unfortunately, the visuospatial sketchpad can perform only temporary storage and rehearsal of the visual information. Thus, to explain about the visual information, it is necessary for the information which is shown as the visual information to be collated with the long-term memory and the auditory information which matches the shown the visual information taken from the long-term memory. Because of this, it is essential to explain that the auditory information is treated. Therefore, the visuospatial sketchpad and the phonological loop are used in the case of the above instance. So, a subsystem is necessary to treat the auditory information and the visual information. The cognitive processing enables collation of the working memory and the long-term memory, and integrates various types of information, such as auditory information and visual information, because there is an episodic buffer. In addition, it is the central executive which performs the role of operating the information. The subsystem is the central existence of the working memory. The working memory is based on the dual storage model which was suggested by Atkinson and Shiffrin. The dual storage model includes a role of the operation of information as part of the functions of the store (Yotaro, 1995). However, operation of the information is realizes that the store and the operation of the information are independent systems due to the development of brain science and neurophysiology because operation of the information is mainly carried out in the frontal lobe and the store and the operation of the information are processed in different places of the brain. So, the working memory model which was suggested by Baddely created a new subsystem. It has the role of operation of the information. Generally, the operation of the information is classified into three components. Those are the abilities to focus on certain information, to shift from the information that attention is paid to new information, and to divide attention into different information. Although, operation of the information has quantitative limit, and it is said that the operation of the information is able to process 4±1 chunks (Cowan, 2001). The chunk is a unit of the lump of the information. There are not enough attention resources to pay attention to the phonological loop when attention is focused on the visuospatial sketchpad, because the central executive has only limited resources to process the information. Thus, processing on the phonological loop is insufficient, and the opposite is assumed. It is forecasted that the cognitive processing is caused interferences, such as the ones above, by limited attention resources.

SRK Model The human cognitive behaviors were classified into three levels by J.Rasmussen (Akinori and Mitsuhiko, 2008). These levels are skill based, rule based, and knowledge based (see Table 1).

Table 1: SRK Model

Table available in the original PDF paper.

Humans recognize the outside information, and a reaction is performed by them. Then, the quantity consumed of the attention resources differs by the degree of the skill of the performer. The customary behaviors cause automatically action and conclude without awareness. So, the attention resources are not needed so much. On the other hands, the first behaviors have to use a lot of attention resources to use all knowledge.

THE EXPERIMENT METHOD The dual task method was carried out in this research to reveal the cognition of the visual information. The signal detection task was carried out in the primary task as the original simulated driving task, and an addition task and a Japanese word chain game task were carried out in the secondary tasks as the conversation on a cell phone using a hands-free device. The addition task is performed without using long-term memory, but the Japanese word chain game task is performed by using long-term memory. Therefore, it is thought that the Japanese word chain game task is nearer the cognitive processing of real conversations than the addition task. In addition, a subjective evaluation about mental stress was examined by NASA-TLX when each task ended (Human Performance Research Group NASA Ames Research Center).

Subjects Twelve undergraduates from Tokai University (eight male, four female) without visual disorders participated in this experiment.

Experiment Conditions In this experiment, only the primary task was called the control condition, the condition which performed the primary task and the addition task was called the addition condition, and the condition which performed the primary task and the Japanese word chain game task was called the conversation condition.

The Primary Task The signal detection task was made in the image of a traffic light (see Figure2). The signals such as a traffic light are randomly displayed on the screen (see Figure 2-1). Then, the subject cannot predict what color the signal will be displayed because the color of the signals is randomized. The size of signals changes with three phases on the constant interval (see Figure 2-1, 2-2, 2-3). The signals were created in order to become gradually large with three phases to imagine the situation that a car approached to a traffic light.

Figure 2. Signal Detection Task
Original PDF Figure 2.

The “Page-Up Key” was set as an accelerator, and the “Page-Down Key” was set as a brake. The subjects were required to press the “Page-Down Key” as a brake when the biggest signal is red (see Figure 2-3). The biggest signal disappears from the screen after a short amount of time. If the subjects could confirm that the biggest signal disappeared from the screen, the subjects who pressed the “Page-Down Key” when the biggest signal was red were required to press the “Page-Up Key” to accelerate. So, when the sizes of the signals are Figure 2-1 and Figure 2-2, and when the color the biggest signal is green or yellow, operating the key is not required.

The Secondary Task Conversations between subject and experimenter were carried out in the secondary task. The addition task and the Japanese word chain game task were carried out as the conversation. In the addition task, the randomized single numbers were prepared, and every number was read by the experimenter after a short amount of time passed. This task required the subject to sum up single numbers and tell about it to the experimenter when the two figures were read. However, the addition task required the subject to add the next problem without restarting it even if the number which was told to the experimenter was not right. The Japanese word chain game task was carried out with the subject and the experiment by the Japanese rule. The task was restarted from the experimenter when it was stopped by rules violated along the way.

Experiment Procedure This experiment had four phases. The first phase carried out the practice of the primary task until the performance of the task was equal to the skill based action form. Then, the control condition, the addition condition and the conversation conditions were carried out for 10 minutes per condition in phases other than the first phase. These conditions were counterbalanced. To exclude the influence of the exhaustion, a break time of 10 minutes was provided when each task concluded. Moreover, the subjective evaluations were examined about the mental stress when each task concluded.

Measurement Item of the Performance of the Primary Task In the signal detection task, the reaction time, the mistake of the reaction, and the oversight of the signals were acquired as measurement items. When the biggest signal was red, the reaction time was defined as from the time that the biggest red signal was indicated on the screen to the time that the “Page-Down Key” was pressed. In addition, if the “Page-Up Key” was pressed when the biggest signal disappeared from the screen, the reaction time was defined as from the time that the biggest signal disappeared from the screen to the time that the “Page-Up Key” was pressed. If the key operations were wrong (e.g. When the “Page-Down Key” needed to be pressed, the “Page-Up Key” was pressed.), the wrong key operations were named the mistake of the reaction. Also, if the reaction time was over 2,000 milliseconds when the key operations were needed, the subject was judged to have overlooked the signal.

Measurement Item of the Subjective Evaluations The subjective evaluations were surveyed by the NASA-TLX. The NASA-TLX has six number lines which have low or high (good or bad) at both ends. It required the subject to mark the subjective impression about the six items of the Rating Sheet (see Figure 3). The six items are defined, and indicated in Table 2. In the ordinary NASA-TLX, it is necessary to choose which of the items is more important to the experiment by the paired comparison method. Then, those informations are used to calculate the weighted rating of the six items. After the questionnaire survey is carried out, each score of the six weighted items are summed and the total score is divided by fifteen to calculate the weighted workload (: WWL). However, a high correlation between WWL and RTLX was showed in a previous study and the correlation coefficient was 0.971 (Shinji and Masaharu, 1993). Therefore, RTLX was used in this experiment. The paired comparison method did not need to use RTLX. So, it was expected that the RTLX lightens the burden of the subject.

Table 2: NASA-TLX Rating Scale TITLE DESCRIPTIONS

Table available in the original PDF paper.

MENTAL DEMAND

How much mental and perceptual activity was required(e.g. thinking, deciding, calculating, remembering, looking, searching, etc) ? Was the task easy or demanding, simple or complex, exacting or forgiving?

PHYSICAL

DEMAND

How much physical activity was required (e.g pushing, pulling, turning, controlling, activating, etc)? Was the task easy or demanding, slow or brisk, slack or strenuous, restful or laborious?

TEMPORAL

DEMAND

How much time pressure did you feel due to the rate or pace at which the tasks elements occurred? Was the pace slow and leisurely of rapid and frantic?

PERFORMANCE

How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)? How satisfied were you with your performance in accomplishing these goals?

EFFORT How hard did you have to work (mentally and physically) to accomplish your level of performance?

FRUSTRATION

LEVEL

How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?

Figure 3. Rating Sheet

Figure available in the original PDF paper.

Experiment Environment This experiment was carried out in a laboratory that has a behavior room and a behavior observation room. The subject in the behavior room was observed by the experimenter in the behavior observation room. The conversation between the subject and the experimenter was carried out by a microphone and the speaker of the laboratory. The performance of the experiment was recorded by a video camera of the behavior room.

EXPERIMENTAL RESULTS Reaction Time

The analysis of variance of the three factors was carried out about the reaction time. The result showed Figure 4, and was recognized as a significant difference (F(2,35)=18.3969 ,P<0.01). In addition, multiple comparisons were carried out and the results indicated significant differences between all factors.

Mistake of Reaction The analysis of variance of the three factors was carried out about the mistake of the reaction. The result showed Figure 5, and was not recognized as a significant difference (F(2,35)=1.9617,P>0.10).

Oversight of Signals The analysis of variance of the three factors was carried out about the oversight of the signals. The result showed Figure 6, and was recognized as a significant difference (F(2,35)=2.5034 ,P<0.10). In addition, multiple comparisons were carried out and the results indicated significant differences between the control condition and the conversation condition.

Subjective Evaluations The analysis of variance of the three factors was carried out about the oversight of the signals. The result showed Figure 7, and was recognized as a significant difference (F(2,35)=9.6040 ,P<0.01). In addition, multiple comparisons were carried out and the results indicated significant differences between the control condition and the addition condition, and also between the addition condition and the conversation condition.

Figure 4. Reaction Time

Figure available in the original PDF paper.

Figure 5. Mistake of Reaction
Original PDF Figure 5.
Figure 6. Oversight of Signals
Original PDF Figure 6.
Figure 7. Subjective Evaluations

Figure available in the original PDF paper.

DISCUSSION

Reaction Time In the addition task, the auditory information is presented by the reading of numbers. When the auditory information is presented, the information is stored in the phonological loop for a temporary amount of time. The first presented number has to be maintained to prevent void at the phonological loop until the second number is read, because the two presented numbers have to be summed. The two presented numbers are summed at the phonological loop when the second number is presented. Then, the two presented numbers before being summed have to be maintained by the articulatory rehearsal component, because the summed number has to be confirmed whether correct or not when it is summed. The summed number is confirmed whether correct or not when the two presented numbers are summed, and it is spoken when it is judged at the correct summed number. The two presented numbers are summed again by the auditory information of the phonological loop when the summed number is not correct. This processing is series of processes of the addition task. The information is not input to the visuospatial sketchpad by the addition task when the process is accomplished such as the above. Therefore, it is not thought that various visual information is input to the visuospatial sketchpad interferes with each other and the cognition of the visual information is interfered by it. Thus, cognition of the visual information may be hindered if the visual information which differs from the main visual information is not inputted to the visuospatial sketchpad. This is because, the difference in the reaction time between the control condition and the addition condition, which is indicated by Figure 4. It is assumed that the central executive contributes to harmful interferences of cognition of the visual information, because humans have limited attention resources. When the auditory information is presented, it is thought that the central executive focuses on the phonological loop to process the auditory information, or the central executive allocates the attention to the phonological loop and the visuospatial sketchpad to process the auditory information and the visual information. The former presumes that the ability of the operation of the visual information is degraded and the cognitive processing of the visual information is not able to be performed smoothly because the attention is focused on the phonological loop and the attention becomes weaker to the information of the visuospatial sketchpad. So, a delay in reaction time was caused. On the other hand, it is conceivable in the latter that the ability of the operation of both the auditory information and the visual information is degraded and the cognitive processing of both the auditory information the visual information is not able to be performed smoothly. Because of this, the subjects tried to process both the primary task and secondary task, and their attention was allocated to both the phonological loop and the visuospatial sketchpad. The primary task is relatively simple and easy. Moreover, the subjects practiced the activity enough times before starting the experiment. So, there are few attention resources required by the primary task, because the primary task is near skilled base. The work of the skilled base and smooth driving by the driver who is used to driving are the same work level. The operation is able to be performed subconsciously when the stimulation is shown. However, in this study, it was indicated that even the work of the skilled base was interfered with easily when the attention was focused on the phonological loop and the attention was allocated to both the phonological loop and the visuospatial sketchpad. To confirm which factor interfered with the reaction, the primary task and the secondary tasks have to be examined by a time series analysis. These are future subjects. The conversation condition is presented by auditory information similar to the addition condition. The presented information is input to the phonological loop, and is stored for a temporary amount of time. The presented information is forgotten after a temporary amount of time passes unless it is maintained by the articulatory rehearsal component. So, the presented information is maintained by the articulatory rehearsal component similar to the addition condition. The point where the interaction between the phonological loop and the long-term memory is performed by the information of the phonological loop is unlike the addition condition. Because the conversation condition is the Japanese word chain game task, a word which is read by the experimenter has to be maintained by the articulatory rehearsal component and a new word is read based on the end of the word. However, according to the working memory model, each new word must be taken from long-term memory, which has the role as the persistent memory storage. Each new word must match with the condition (e.g. to confirm whether or not the end of a word and the first character of a new word is the same character). Then, only the end of a word in the phonological loop is transmitted to the episodic buffer to search and collate in the long-term memory. For each new word, the end of the word and the first character of a new matching word are taken from long-term memory and stored in the episodic buffer. Then, the new word is transmitted to the phonological loop to speak it. It is thought that the judgment about whether the word has already been spoken or not is carried out in both the episodic buffer and the phonological loop. The primacy effect and the recency effect were confirmed by Glanzer when the information is

shown serially is stored. The probability rises against the information that is presented in first of a series, because it is possible to maintain it by the articulatory rehearsal component many times. The information that was just shown confirms the recency effect because it is stored to short-term storage. In this case, it is assumed that a new word, which is taken from long-term memory, and a known word, which is stored in long-term memory, are collated at the episodic buffer and a new word, which is taken from long-term memory, and a known word, which is stored in short-term memory, are collated at the phonological loop. When a new word is judged to not be a known word, it is spoken. Thus, the reaction time is slower than the addition condition, because the conversation condition is a more complicated cognitive processing than the addition condition, and the attention resources are used more than the addition task. It is suggested that the delay in cognition of the visual information is caused by the interaction of the auditory information, because it is indicated that even skilled base work which does not require the attention so much delays the cognition of the visual information.

Mistake of Reaction The mistake of the reaction was not recognized as a significant difference. It is thought that the correct reaction is carried out if the signals are recognized, because the primary task is a skilled base work and simple work. From the results of the reaction time and the mistake of reaction, the delay in cognition of the visual information may be caused by the interaction of the auditory information, even if the cognition work of the visual information is equal to the skilled base work. However, it is possible that the person carries out relatively right judgments if the information is recognized.

Oversight of Signals The oversight of signals indicates significant differences between the control condition and the conversation condition. The condition which indicates which cognition of visual information is not able to be carried out is caused by presentation of auditory information. In this study, the eye movement data was not captured. So, the possibility of oversight caused by shifts in line of vision is not able to be completely denied. It is thought that the complicated cognitive processing, like the Japanese word chain game task, uses a lot of attention resources. The information beyond the attention resources is forgotten without the cognitive processing if a lot of attention resources are used. When the attention is not allocated to the signals, the signal of the visual information is input to sensory memory, but the cognitive processing is not carried and is forgotten because the attention is not able to be allocated to the signal (see Figure 8). Therefore, it is conceivable that a condition in which perception is able to be carried out but not cognitive processing is caused. It is indicated that the cognition of the visual information interferes with the auditory information.

NO ATTENTION

ATTENTION

Figure 8. Cognitive Processing https://openaccess.cms-conferences.org/#/publications/book/978-1-4951-2104-3
Original PDF Figure 8.

Physical Ergonomics I (2018)

Subjective Evaluations The subjective evaluations indicated significant differences between the control condition and the addition condition. It is thought that the conversation condition needs more cognitive processing than the addition condition does. However, the subjective evaluations did not indicate significant differences between the addition condition and the conversation condition. These results were acquired because the addition condition was simpler to work and a large mental workload was felt. On the other hand, it is assumed that the mental workload diminishes, because the conversation condition is performed like a game and there are few boring impressions. The RTLX of the conversation condition is about the same as the addition condition. Although, according to the reaction time and the oversight of the signals performances, it is indicated that cognitive processing is hindered by the conversation condition. It is probable that the cognitive processing is complicated even if the subjective mental load for the cognitive processing is estimated to be small. A correlation is not seen in the subjective evaluations to the cognitive load or the real cognitive load. It is suggested that the person does not have abilities which correctly estimate the real cognitive load.

CONCLUSION

This research revealed whether the conversation with a hands-free device interferes with the cognition of visual information. In this experiment, it was indicated that the delay of the cognition of the visual information and the oversight of the signals were caused by interactions with auditory information. In addition, a correlation between the subjective evaluations to the mental workload and the performance of the cognitive processing is not recognized, and it is probable that the decline of the cognitive processing is recognized even if the subjective mental load for the cognitive processing is estimated to be small. It is suggested that the person cannot correctly judge the real cognitive load. Thus, it is thought that the cognition of the visual information is hindered by conversations using hands-free device because it was indicated that the cognition of the visual information was interfered sufficiently by the interaction of the auditory information. This study was not examined about the allotment condition of the attention resources, because the performances of the primary task and secondary task had to be analyzed by a time series. Additionally, eye movement data had to be acquired about the oversight of signals. These are points for future studies.

REFERENCES

Akira Miyake, Satoru Saito (2001), “the Japanese Journal of Psychology”, Current trends and future directions in working memory research, Volume 72 No.4, pp. 336-350 Parks. A. M. (1991), “Contemporary ergonomics: Proceedings of the Ergonomics Society annual conference”, Drivers business

decision making ability whilst using carphone, pp427-432 Alan Baddely (2012), “seishinshobo”, Working Memory Thught and Action Yotaro Takano (1995), “University of Tokyo Press”, cognitive psychology 2 memory Cowan N. (2001), “Behavioral and Brain Sciences”, The magical number 4 in short-term memory: A reconsideration of mental storage capacity, Volume 24 No.1, pp87-114 Akinori Komatsubara, Mitsuhiko Krashima (2008),” Asakura Publishing”, Management Ergonomics Human Performance Research Group NASA Ames Research Center, NASA TASK LOAD INDEX(TLX) Paper and Pencil

Package Shinji Miyake, Masaharu Kumashiro (1993), “Japan Ergonomics Society”, Subjective mental workload assessment technique – An introduction to NASA-TLX and SWAT and a proposal of simple scoring methods-, Volume 29 No.6, pp399-408

Cover of the proceedings volume Advances in Physical Ergonomics and Human Factors: Part I

Explore the complete digital volume

Advances in Physical Ergonomics and Human Factors: Part I

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-4951-2104-3

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →