Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

How does it work? Collecting mental models for the user-centered design of assistance systems: a construction machinery case study

1 Technical University of Munich, TUM School of Engineering and Design, Chair of Ergonomics, Garching b. München, 85747, Germany ↗
2 Technical University of Munich, TUM School of Engineering and Design, Institute of Automation and Information Systems, Garching b. München, 85748, Germany ↗
3 BAUER Maschinen GmbH, Schrobenhausen, 86529, Germany ↗
Publication year
Digital Object Identifier10.54941/ahfe1004505

Abstract

This paper proposes a three-step procedure to systematically include human operators in the development and design process of an assistance system for complex construction machinery. The central element of the procedure is the collection of mental models to gain validated information on the context of use and the human-machine interaction from the operator's perspective. The procedure was used in a case study to develop a control assistant for hydraulic grab carriers. The results demonstrate the feasibility of the proposed procedure.

INTRODUCTION

The widespread influence of automation spans multiple domains, including civil engineering and excavator systems, producing greater functionality and complexity. The indispensable role of human operators today and in the future requires a human-centered approach throughout the design, imple- mentation, and evaluation phase when adding automation or assistance systems to established machinery.

Underground construction machinery is a complex system that poses strong requirements on human operators. The context in which such machin- ery is used is often characterized by high time pressure, harsh environmental working conditions, and uncertainty about the composition of the ground, which makes it challenging to automate. Operators of underground construc- tion machinery acquire a broad skill set that goes far beyond the basic control of the machinery’s functionalities. It takes years of practice to become an expert machine operator, develop confidence in the machine’s control, and

© 2024. Published by AHFE Open Access. All rights reserved. 412

How Does It Work? Collecting Mental Models for the User-Centered Design 413

find the correct parameters according to the specific conditions on the con- struction site. However, there are opportunities for automation systems to support and assist the operator in specific tasks, reducing workload and increasing the overall efficiency of the human-machine system if designed carefully.

This paper proposes a procedure using mental models to systematically include human operators in the development and design process of an assistant system for complex construction machinery to obtain validated information on the context of use and the human-machine interaction from the operator’s perspective. In addition, we present a case study in which we followed the proposed procedure. An exemplary picture of construction machinery, as considered in this work, is shown in Figure 1.

Figure 1: Exemplary construction equipment: hydraulic grab carrier (BAUER Maschinen GmbH).
Original PDF Figure 1:

MENTAL MODELS: WHAT, WHY, AND HOW?

Furlough and Gillan (2018) define mental models as mental representations of the external world that humans use while interacting with the environment and systems. According to Jones et al. (2011), mental models are a cognitive structure that forms the basis for thinking, decision-making, and behavior. Mental models can be further differentiated into perceptive or causal mod- els (Johnson-Laird, 1983) and individual or collective mental models (Jones et al., 2011).

When designing systems with human-machine interfaces, Norman (1986) differentiates between three models: the conceptual model of the user, the user’s conceptual model, and a physical image of the system. The two

conceptual models contribute to the image of the system. Therefore, the developer’s task is to provide the user with adequate information about the system image to make the product comprehensible and usable. Furthermore, the mental model must match the image of the system. Otherwise, it could cause problems during use and negatively affect user experience (Norman, 2013). Mental models also help to understand user motivation, requirements, and identify user knowledge or knowledge gaps (Nielsen, 2010; Young, 2008). The models are based on personal life experiences, perceptions, and under- standing of the world and, therefore, are part of learning processes and form the mechanism by which new information is filtered and stored (Jones et al., 2011). It is essential that users effectively communicate these models in real- life situations they encounter. However, individuals have only partial access to their mental models and can express their content only in a limited way. Interpretation and representation thus provide second-order mental models (Schaffernicht and Grösser, 2015). There is a wide range of methods to gener- ate mental models, such as interviews, questionnaires, think-aloud tasks, task analysis, verbal protocol, card sorting, and training material (Jones et al., 2011; Jones et al., 2014; Laukkanen and Wang, 2015; Schaffernicht and Grösser, 2015).

Analyzing the collected data provides a basis for representing mental mod- els. The representation often takes the form of cognitive maps, a diagram that visualizes elements perceived to be essential and the relationships between them (Schaffernicht, 2017). Visualization of mental models enables the com- munication and validation process. The method of data collection and the form of representation strongly influence each other and cannot be separated from each other (Kane and Trochim, 2009; LaMere et al., 2020). Depend- ing on the purpose of the representation, these visualizations take different forms (Schaffernicht, 2017). This procedure focuses on determining the cur- rent conceptual framework of human operators of construction machinery; thus, the chosen visualization method is a concept map.

Concept Mapping Concept mapping focuses on organizing and representing a group’s knowl- edge about a domain (Kane and Trochim, 2009). Concept maps depict descriptive and causal relationships where concepts are objects or events, and the connections are static or dynamic (Cañas, 2009a, 2009b). The for- mulation of concept maps helps realistically map the operators’ language and terminology to be investigated. The inherent property of a sentence-like structure contributes to the ease of interpretation of the reader without prior knowledge (Cañas, 2009b). Figure 2 shows a sample concept map, where ‘Concept Maps’, ‘Organized Knowledge’, ‘Effective Teaching’, etc. are con- cepts and ‘helps to answer’, ‘includes’, ‘needed to answer’ etc. are linking words that build relationships between concepts (Cañas and Novak, 2009a).

Concepts are essential building blocks representing underlying principles, thoughts, and beliefs and thus play an important role in all aspects of cogni- tion (Goguen, 2005). A concept map is a graphical representation of a set of

How Does It Work? Collecting Mental Models for the User-Centered Design 415

statements about any topic. The hierarchical structure is based on the notion that each domain has a hierarchy of concepts. The concept map takes this into account by placing the most general concepts at the top of the hierar- chy and the more specific, less general concepts hierarchically below (Cañas and Novak, 2009b). This nature of concept maps helps represent the mental model regarding construction machinery in general and depth.

Figure 2: Concept map showing important features of a concept map (Cañas and Novak 2009a).
Original PDF Figure 2:

PROCEDURE Especially in the case of highly complex machinery and application areas, such as underground construction operations, a correct user mental model is evident to avoid damages and ensure the efficient operation of these power- ful technical systems. According to the human-centered design process (ISO, 2020), the first step focuses on understanding the personas in contact with the machinery, the task to be solved, and the context of use, including the human-machine interaction. This work proposes a process to obtain vali- dated information on the use context and the human-machine interaction from the operator’s perspective. We use mental models and a collective con- cept map to capture the technical aspects of human-machine interaction and the user’s intentions when using multiple machine interfaces. Mental models are a powerful tool for comprehending the cause-and-effect relationships of operators, perceived advantages, and user requirements. Concept maps offer the possibility of visualizing the relevant system elements and their relations from different perspectives and levels of detail.

The process is structured in three steps: data collection, aggregation and abstraction, and validation.

In the first step, qualitative semi-structured interviews with representatives of relevant user personas serve as an effective explorative technique to gain a

thorough understanding of users, the specific task users need to solve in coop- eration with the machinery, the human-machine interaction, and the context of use. In addition, interviews help identify essential aspects of the system design process, such as affordances, user expectations, and system and inter- action requirements from the user’s perspective. Thus, interviews are divided into a deductive and an inductive part. The deductive part consists of a user journey. In a user journey, the interviewees describe precisely which steps they take to complete a specific task and share their experiences with the interac- tion. The technique helps to identify the current user workflow at a high level of detail, understand the context of use, affordances, and user expecta- tions, and reveal areas of improvement. The inductive part is used to collect structured information on human-machine interaction, identify relevant sys- tem elements for users, and understand the directions and functionalities of the relationships (links) between these elements as a structured representa- tion of the operators’ understanding of the components, functionalities, and their control of the system. The literature suggests the creation of individual concept maps for individual mental models. However, we propose to skip this intermediate step, as the scope of this procedure is to gain a thorough understanding of the human-machine interaction among all users instead of an in-depth comparison of different user personas.

In the second step, individual descriptions are aggregated and abstracted. Aggregating the individual mental models leads to a near-complete structured description of the perceived cause-effect relations and can identify differences between users. Understanding and analyzing the reasons behind these dif- ferences is important, as they can offer valuable insights for the following interaction design. The aggregated individual mental models are formalized in an iterative adaptation and extension process of the concept map. The specific elements of the map are derived from user descriptions. According to the works of (Laukkanen and Wang, 2015; Schaffernicht and Grösser, 2015), the primary criterion for the inclusion of an element is the achievement of a threshold based on frequency among all participants. We propose a threshold of 50 %; hence, if an element is mentioned in more than 50 % of the individ- ual descriptions, it is to be included in the aggregated concept map. Further, elements that meet one of the following criteria should also be included: the element is an objectively verifiable fact, and/or the element is deemed essen- tial to communicate implications for the assistant design. The result of the second step is a collective mental model visualized by a single concept map.

Finally, the third step contains the validation of the collective mental model. Representatives of relevant user personas review the collective concept map in a second round of user interviews to ensure the formal correctness of the content and to obtain feedback. Depending on the size and level of detail considered in the concept map, it might be practical to present a con- densed version of the map to the interviewees. When doing so, it is advisable to retain the elements essential to the concept map’s logic. In addition, ele- ments and links whose importance and correctness are considered uncertain should remain in the short version. After this final step, the collective con- cept map can be used as a validated representation of the mental model of the human-machine interaction and the technical cause-effect relationships.

How Does It Work? Collecting Mental Models for the User-Centered Design 417

The procedure is shown in Figure 3.

Figure 3: Procedural steps.
Original PDF Figure 3:

CASE STUDY In this case study, we analyzed the human-machine interaction of operators with a BAUER hydraulic grab carrier. We identified machine operators with different experience levels and product managers of the respective machine type as relevant personas for the analysis of the context of use and the human- machine interaction in its current state.

In the data collection step, semi-structured interviews were conducted with five operators of different experience levels (mean age 36.8 years, range 25 – 56 years, all male). Various recommendations for an adequate number of participants can be found in the literature (AlRoobaea and Mayhew, 2014; Bevan et al., 2003; Faulkner, 2003; Lewis, 2006). Due to the scarcity of avail- able personnel in the considered application, a number of five participants was viewed as sufficient, which is supported by (Nielsen, 2000). Because the interviewees worked on construction sites scattered throughout Germany, all interviews had to be conducted by telephone or video calls. Table 1 pro- vides an overview of the characteristics of the interview participants and the interview specifications.

In the second step, the aggregation and abstraction of the individual mental models were performed sequentially. The five interview transcripts were ana- lyzed line by line for explicit (direct mention) and implicit (indirect mention) concept map elements. Due to the uneven number of participants, the thresh- old of 50 % for inclusion of an element in the collective map was adjusted to 60 %. Consequently, an element was only included if it was identified in at least three of the five interview transcripts.

In the third step, the collective mental model was validated by three of the five previous interviewees. Validation interviews were conducted through the online platform zoom™. Participants were guided through the collective con- cept map, asked to look for discrepancies and errors, and were encouraged to give feedback. After the interviews, the concept map was slightly adjusted based on the interviewees’ feedback.

The validated concept map is used as a representation of the operators’ understanding of the cause-effect relationships and the human-machine inter- action. As such, it plays an essential role in the design of an assistance system for machine control.

Table 1. Overview of interview participants.

Table available in the original PDF paper.

Procedural Step

Alias Experience [years]

Interview Format

Self-assessed experience level

Interview Duration [min]

Alfa 31 Expert Phone 59 Data Collection Bravo 1 Novice Phone 46 Data Collection Charlie 10 Novice Zoom 79 Data Collection Delta 6 Novice Phone 60 Data Collection Echo 10 Expert Phone 43 Data Collection Alfa 31 Expert Zoom 30 Validation Charlie 10 Novice Zoom 49 Validation Delta 6 Novice Zoom 39 Validation

CONCLUSION

This paper proposes a three-step procedure using mental models to system- atically include human operators in the development and design process of an assistance system for complex construction machinery to obtain validated information on the context of use and human-machine interaction from the operator’s perspective.

The procedure was used in a case study to develop a control assistant for hydraulic grab carriers. The results demonstrate the feasibility of the pro- posed procedure. Due to the systematic approach and early inclusion of human operators, the authors were able to build the design phase on vali- dated information on the operators’ understanding of technical functionality and machine control, as well as their expectations. To evaluate and compare the benefit of the process with other approaches, more research should be conducted in the future.

ACKNOWLEDGMENT

This work was supported by the Bayerische Forschungsstiftung (BFS) through the project ‘Maschinenführer-zentrierte Parametrierung von Arti- ficial Intelligence für eng gekoppelte, verteilte, vernetzte Steuerungssysteme (OpAI4DNCS)’.

How Does It Work? Collecting Mental Models for the User-Centered Design 419

REFERENCES

AlRoobaea, R. and Mayhew, P. J. (2014) ‘How many participants are really enough for usability studies?’, 2014 Science and Information Conference. London, UK, August 27-29, 2014, IEEE, pp. 48–56. BAUER Maschinen GmbH BAUER GB 80 S. Bevan, N., Barnum, C., Cockton, G., Nielsen, J., Spool, J. and Wixon, D. (2003) ‘The

“magic number 5”’, CHI’03 extended abstracts on Human factors in computing systems. New York, NY, USA, ACM Press, pp. 698–699. Cañas, A. J. (2009a) ‘What are linking words?. From a concept mapping perspective’. Available at https://cmap.ihmc.us/docs/linkingwords.php (Accessed 3 October 2023). Cañas, A. J. (2009b) ‘What are propositions?. From a concept mapping perspec-

tive’. Available at https://cmap.ihmc.us/docs/proposition.php (Accessed 3 October 2023). Cañas, A. J. and Novak, J. D. (2009a) ‘What is a concept map?’. Available at https://cmap.ihmc.us/docs/conceptmap.php (Accessed 3 October 2023). Cañas, A. J. and Novak, J. D. (2009b) ‘What is a concept? From a concept map-

ping perspective’. Available at https://cmap.ihmc.us/docs/concept.php (Accessed 3 October 2023). Faulkner, L. (2003) ‘Beyond the five-user assumption: benefits of increased sample sizes in usability testing’, Behavior Research Methods, Instruments, & Computers, vol. 35, no. 3, pp. 379–383. Furlough, C. S. and Gillan, D. J. (2018) ‘Mental Models: Structural Differences and

the Role of Experience’, Journal of Cognitive Engineering and Decision Making, vol. 12, no. 4, pp. 269–287. Goguen, J. (2005) ‘What Is a Concept?’, Springer, Berlin, Heidelberg, pp. 52–77. International Organization for Standardization (2020) ISO 9241-210:2019: Ergonomics of human-system interaction – Part 210: Human-centred design for interactive systems (ISO 9241-210:2019); German version EN ISO 9241- 210:2019, Berlin: Beuth Verlag GmbH. Johnson-Laird, P. N. (1983) ‘Mental models: Towards a cognitive science of language, inference, and consciousness’, Harvard Univ. Press. Jones, N. A., Ross, H., Lynam, T. and Perez, P. (2014) ‘Eliciting Mental Mod-

els: a Comparison of Interview Procedures in the Context of Natural Resource Management’, Ecology and Society, vol. 19, no. 1. Jones, N. A., Ross, H., Lynam, T., Perez, P. and Leitch, A. (2011) ‘Mental Models: An Interdisciplinary Synthesis of Theory and Methods’, Ecology and Society, vol. 16, no. 1. Kane, M. and Trochim, W. M. K. (2009) ‘Concept Mapping for Planning and

Evaluation’, Journal of Mixed Methods Research, vol. 3, no. 1, pp. 87–89. LaMere, K., Mäntyniemi, S., Vanhatalo, J. and Haapasaari, P. (2020) ‘Making the most of mental models: Advancing the methodology for mental model elicita- tion and documentation with expert stakeholders’, Environmental Modelling & Software, vol. 124, p. 104589. Laukkanen, M. and Wang, M. (2015) ‘Comparative causal mapping: The CMAP3

method’, London, New York, Routledge, Taylor & Francis Group. Available at https://www.taylorfrancis.com/books/mono/10.4324/9781315573038/compar ative-causal-mapping-mingde-wang-mauri-laukkanen. Lewis, J. R. (2006) ‘Sample sizes for usability tests: mostly math, not magic’, in interactions, 6th edn, pp. 29–33.

Nielsen, J. (2000) ‘Why you only need to test with 5 users’. Available at https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ (Accessed 10 April 2023). Nielsen, J. (2010) ‘Mental models and user experience design’. Available at

https://www.nngroup.com/articles/mental-models/ (Accessed 10 October 2023). Norman, D. A. (1986) ‘Cognitive Engineering’, in Norman, D. A. and Draper, S. W.

(eds) User Centered System Design, Boca Raton, CRC Press, pp. 31–62. Norman, D. A. (2013) ‘The Design of Everyday Things’: (Rev. and expanded ed.), Basic Books. Schaffernicht, M. F. G. (2017) ‘Causal attributions of vineyard executives – A mental

model study of vineyard management’, Wine Economics and Policy, vol. 6, no. 2, pp. 107–135. Schaffernicht, M. F. G. and Grösser, S. N. (2015) ‘Capturing managerial cognition in Chilean wineries: hardening a new method to elicit and code mental models of dynamic systems’, Proceedings of the 33rd International Conference of the System Dynamics Society. Young, I. (2008) ‘Mental models: aligning design strategy with human behavior’,

Brooklyn, New York, Rosenfeld Media.

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-95-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →