Masaya Ando, Yasunobu Ito (2022). Conceptual Change in Human-centered Design by Artificial Intelligence System. In: The Human Side of Service Engineering. AHFE Open Access, vol. 62, pp. 143–151. https://doi.org/10.54941/ahfe1002552
It has been pointed out that the development of artificial intelligence can transform the relationship between systems and people fundamentally. Human-centered design (HCD) is a concept that realizes a human-centered system by grasping user usage and deriving user requirements. This design concept is still important and is effective when systematizing existing operations. However, when a user's advanced work can be replaced by artificial intelligence, the relationship between the user and the system changes dramatically, but the direction has not been clarified. In this study, interviews were conducted with service providers on the site using artificial intelligence and discussed the necessity of rebuilding the meaning and significance of the operations and actions of the user using the system.
Artificial intelligence (AI) technology has made remarkable progress in recent years. Its core technology, deep learning, has a wide range of applications, including image and video generation, natural spoken dialogue, anomaly detection, translation and text understanding, security, and drug discovery. Its notable achievement for certain well-defined tasks, such as image reco- gnition, is learning on labeled data from the formal world which has yielded higher accuracy and better performance than humans. These technologies are already being applied to systems and services for specific purposes, and are becoming pervasive in society.
While there are growing expectations for the various benefits that AI technology brings to human society, various social issues are pointed out, such as privacy, ethics, decision-making autonomy, and the impact on social systems. For this reason, efforts are being made at the global level to esta- blish ethical guidelines for AI development (Fukuzumi et al., 2019). For example, the Organization for Economic Co-operations and Development (OECD) formulated the “OECD Principles on Artificial Intelligence”, which was adopted by 42 countries in May 2019 (OECD, 2019). In Japan, “Social Principles of Human-Centric AI” was published by the Cabinet Office in March 2019 (Cabinet Office in Japan, 2019). As the title of the guideline by the Cabinet Office indicates, it is essential that AI development and its social implementation be “Human-Centric”. However, the meaning of the
term “human-centric” itself is not defined. Though based on the content of the principles, “human-centric” can be understood to mean respect for basic human rights and diversity, fairness, and justice.
On the other hand, in the research field of Human Computer Interaction (HCI) or Information System (IS), Human-Centered Design (HCD) has been studied and practiced for a long time. In particular, the International Orga- nization for Standardization (ISO) published ISO 13407, an international standard for human-centered design, in 1999, which has been succeeded and developed as ISO 9241-210 (2019).
Therefore, in the field of system development, human-centered reaso- ning has been widespread and practiced. However, there is a possibility that the conventional way of human-centered thinking cannot be applied to AI systems, and this can be interpreted as the reason why new principles and guidelines are presented. In other words, there is a need to reexamine HCD, especially the concept of human-centeredness, with AI systems in mind.
In this paper, we assume AI systems from the standpoint of HCD, and exa- mine the focus points necessary to realize human-centeredness. To this end, we conducted interviews with developers of services where AI systems have been introduced. In the interviews, we aim to understand the relationship between the users and the system, whether there are any differences compa- red to the conventional relationships between users and the system and what kind of measures are being taken in dealing with the differences. Against this backdrop, we discuss the human-centered approach in AI systems.
LITERATURE REVIEW
Human-Centered Thinking in Existing HCD ISO 9241-210 (2019) does not define the term “human-centered” per se, but defines “human-centered design” as follows:
An approach to systems design and development that aims to make intera- ctive systems more usable by focusing on the use of the system and applying human factors/ergonomics and usability knowledge and techniques (ISO 9241-210, 2019) The well-known ISO HCD diagram depicts the interdependence of the four activities (Figure 1).
As stated in the standard, HCD activities can start at any stage but as Figure 1 shows, “Understand and specify the context of use” plays an impor- tant role in the early stages of systems development. The “context of use” is defined as “users, tasks, equipment, and the physical and social environments in which a product is used.” It is a concept that takes on a relatively broad view of the situations and contexts in which users and stakeholders interact with the systems.
A particular note of ISO HCD is the importance of the context of use that is mentioned throughout the standard. Humans are still the main element in designing but incorporating work processes and work environment in the design is a concept common to overall systems design in ergonomics (ISO 26800, 2011).
Conceptual Change in Human-centered Design by Artificial Intelligence System 145
Figure 1: Interdependence of human-centered design activities (ISO 9241-210, 2019).
To view context of use not just as human characteristics but as intera- ctions with the environment and people, is a common approach in other related research. For example, in the law of behavior (field theory) presented by Lewin, K., a social psychologist, the structure of a field in which people and the environment are interrelated is defined as a “living space,” where behavior (B) calculates a function for personal characteristic conditions (P) and environmental conditions (E) (1) (Lewin, 1951).
B = f(P · E) (1) ISO’s HCD concept does not view humans as a single entity with its own characteristics, but as an entity that interacts with the environment, with the focus on usage. In other words, human-centeredness is a systematic approach to design that focuses on understanding the usage situation, including the user.
Giacomin’s Approach to Human-Centered Design In the process of establishing the ISO HCD standard, it was mainly designed with business use in mind. Even though the current standard does not clearly delineate the scope, it is clear that instrumental systems are envisioned with definite user goals in mind. In contrast, there is a problem with systems which target general consumers as they face diverse interactions with the systems where it is not necessarily perceived as “tools” by consumers (Giacomin, 2014). Giacomin, J., a researcher of HCD, indicates a change in the HCD para- digm in recent years, citing Apple’s products as an example. The paradigm shift is an approach not only limited to the interactions with stakeholders within the framework of existing products, systems, services, and meanings, but a foremost approach where individuals whose contact with the design derive metaphysical meanings.
Giacomin presented the model of the human-centered design pyramid shown in Figure 2, and related the classic rhetorical questions of “Quis (who),” “Quid (what),” “Quando (when),” “Quem ad Modum (in what
way),” and “Cur (why)” to the design considerations. The higher the pyra- mid level, the greater the complexity, according to Giacomin. He describes this diagram as follows: This new interpretation of human centered design is based on a hierarchy which has at its base the scientific facts about human physical, perce- ptual, cognitive and emotional characteristics, followed by progressively more complex, interactive and sociological considerations. At its apex the model contains the metaphysical meaning which individuals form based on contact with the design. In the view which is summarized by the model, the metaphysical meaning, whether pre-existing or still to be created through contact, is considered the key to social acceptance, commercial success, brand identity and business strategy. (Giacomin, 2014)
Giacomin’s human-centered design pyramid focuses on the human-related aspects of ISO usage, and places emphasis on the resulting symbols, commu- nication, discourse, and meaning.
In AI-based systems, interactions with systems are expected to change. It may be possible to grasp the current usage of the system so that changes can be simulated at the preliminary stage of building the system, but it is considered infeasible at present. Therefore, the HCD concept of ISO cannot be applied as it is. On the other hand, Giacomin’s HCD which emphasizes on the “meaning,” can handle new meanings, to this end, it is possible to apply this concept. In the next section, we will review methods on how this can be carried out and examine the direction that should be considered by investigating the changes in the relationship between users and AI systems.
METHOD
AI is being introduced in a variety of actual services. Some stakeholders are likely to experience a different impact on the changes compared to the implementation of conventional information systems. Therefore, we deci- ded to take up only such cases and conduct interviews with developers in order to obtain clues for human-centered thinking in systems that utilize AI technology.
Conceptual Change in Human-centered Design by Artificial Intelligence System 147
Table 1. Investigation targets.
Table available in the original PDF paper.
Company A: Educational material service for cram schools
System provides teaching materials for cram schools that uses AI to diagnose the proficiency level of students, recommends the contents of study that are essentially necessary for students, and provides a tailor-made curriculum.
Company B: Sensors and business systems for nursing homes
Business system for elderly care facilities, provides camera sensors which use AI to analyze resident behaviors and improve the work efficiency of the nursing staff.
The subject of this paper is a system in which AI replaces some kind of human discovery, judgment, or identification. In other words, a typical AI system is one in which the introduction of AI technology causes change in the traditional roles of humans and that of systems. Specifically, we will focus on systems for business use. In this paper, we report on the following two cases. In both cases, the actual state of AI technology is based on machine learning.
Interviews were conducted with the system developers of these services. The developers were not necessarily those in charge of AI technology, but they were representatives related to HCD and UX. The interviews were conducted with one interviewee at Company A and three interviewees at Company B (conducted simultaneously). The interviews lasted about 90 minutes in each case.
The main common contents of the interviews were as follows:
Confirmation of the outline of the services provided
The meaning and significance of using AI in this service field
Customer reaction and response to the introduction of AI-based services
Issues in the development of the services and approaches to HCD and UX design
RESULT
The following is a brief description of the characteristic findings, with focus on the changes in customers as a result of the introduction of AI-based systems.
Company A (Educational Materials for Cram Schools)
Tutors at regular cram schools have a strong desire to “teach” their stu- dents. Our product is a teaching material that aims to help students understand on their own. Before students can start to comprehend, tutors mention the know-how to solve the problem, making students feel as if they “get it”, but as they work on their own, it’s obvious they do not understand at all.
148 Ando and Ito
Our products are designed so that surface-level understanding does not happen. If the AI estimates that the user does not understand something, it goes back to the underlying cause to promote understanding. For exam- ple, if you want to learn high school physics, you may be guided to learn fractions from elementary school curriculum.
We develop the service based on the belief that this kind of learning leads to true comprehension. In fact, the data reflects this result. Therefore, I would like teachers to become supporters of students and encourage them, rather than teach them. We need teachers to resist the urge to teach. To ensure students’ fundamental understanding, I request teachers to believe in our products and offer support to students.
In fact, when student performance improves, teacher attitude changes. The attitude change in teachers can be seen as their role shift from teaching to encouraging and motivating students.
Company B (Sensors and Business Systems for Nursing Homes)
Our system uses a camera as a sensor to detect when a person requiring nursing care (resident of a nursing home) is out of their normal position in bed and sends the image to the staff’s terminal. The developers assu- med that by sending the image, staff would be able to judge whether the residents require their immediate help or not, since the image is clear. How- ever, in reality, it was difficult for them to make the decision. Some staff members just look at the image and do not rush to the scene, while others do not look at the image and rush to the scene every time. This is probably because they do not understand the features of this system correctly.
Some facilities have a clear-cut purpose for using this system. In such faci- lities, this is less likely to happen and promotes staff’s understanding of the system.
The nursing care staff are professionals in terms of knowing the residents’ feelings, physical conditions, and are keen to realize what the residents’ wishes are, but they are not used to designing and improving the operation system, thus they fell short of developers’ expectations.
We would like staff to come up with a process on how to deal with the case when the sensor incorrectly detects abnormality. It is essential for us to think together with the nursing care staff about the kind of roles they should be playing in handling this system. Currently, we are conducting these pre-implementation workshops before introducing the system.
Summary of Case Studies The contents of the interviews with the two companies have common outco- mes. What evolved in the introduction of a new system using AI technology is the change in the relationship between the system and the users (i.e, tutors and nursing care staff), as well as the relationship between the users and the service recipients (i.e, students and facility residents). Especially notable is the relationship between the system and the user because the user is required to play a different role from that of the conventional system. For example, tutors are now expected to encourage students rather than teach them, and
Conceptual Change in Human-centered Design by Artificial Intelligence System 149
Figure 3: Hypothesis of the state of HCD in AI systems.
nursing care staff are now expected to think about the business operations in the system.
Furthermore, in both cases, a common trait was observed in the field where users either resisted or were confused by these role changes. It can be conclu- ded that users do not accept their new roles simply by introducing the system to them, but necessitate the support and encouragement from developers till users foster understanding to change their attitudes toward their new roles.
Creating a new role depends on whether the user can construct a new meaning between the system and the user. Not only does this require a certain amount of time for the user to come to realization, but in some cases, it may also require support and push from the development side to help the user build the meaning.
DISCUSSION
In this paper, we first reviewed the conventional HCD approach to human- centered thinking in AI-based systems, and conducted interviews to company developers that offer AI service systems.
Based on the interview results, we created a hypothesis of how HCD could be approached in AI-based systems as shown in Figure 3.
Conventional systems are designed to help users achieve their business objectives and goals. In Giacomin’s HCD pyramid, relatively lower-order elements are emphasized. In such systems, it is important to understand the current usage of the system, including the users, and to consider what kind of interactions with the systems are desired which are fundamental to human-centeredness.
However, AI-based systems can take over and perform the low-order inten- tions (goals) of users. Under this system, the relationship between the system and the user changes, creating a need for the user to construct a higher-order meaning within the system. This is required of the user after the implemen- tation of the system and should gradually take place. Once the new meaning is constructed, the new role of using the system with AI is recognized.
150 Ando and Ito
Bearing this process in mind, we come to realize that the scope of HCD is not limited to developing AI systems, but also requires the user to build new roles and meanings around how users interact with AI systems. Supporting this process itself is also considered to be an essential activity for realizing human-centeredness.
The efforts of Sunaga among others are rich in suggestions for the ful- fillment of human-centeredness (Sunaga, T., et al, 2014). Sunaga, and their colleagues entered the field of hospitals for the purpose of designing electro- nic medical records, but after six years, they regularly hold workshops where nurses rethink the concept of “the heart of nursing”. Although no electro- nic medical record system has actually been developed, the process of nurses becoming aware of their newly established roles and meanings they are sup- posed to fulfill are being designed. By adding this kind of process, it may be possible to make the relationship between AI-based systems human-centered and designable.
CONCLUSION
In this study, we examined HCD and hypothesized on the human-centered approach to systems development using AI.
Currently, appropriate AI development is an urgent research issue. In this context, it is our intention that the concept of human-centeredness not only be expressed as a term of respect for human philosophy, but as a concrete concept which should be incorporated in the designing process.
In this paper, we have not yet conducted a sufficient number of surveys or held in-depth discussions, but we would like to further our discussions with related parties and present our ideas as guidelines on how human-centered AI development ought to be conceived.
ACKNOWLEDGMENT
This work was supported in part by JSPS KAKENHI Grant Numbers 19H05469/20K20405.
REFERENCES
Cabinet Office in Japan, Council for Social Principles of Human-centric AI (2019), “Social Principles of Human-Centric Artificial Intellengence(AI),”online at: https: //www8.cao.go.jp/cstp/english/humancentricai.pdf. Fukuzumi, S., Jinno, M., Inagaki, K., Maeda, H., Mizukami, T., & Sakura, O.
(2019), “Extraction of New Guideline Items from the View Point of ELSI (Ethics, Legal, Social Issues) for Service Utilized AI–focus on Healthcare Area.” Inter- national Conference on Human-Computer Interaction, LNCS, volume 11570, pp. 482–491. Giacomin, J. (2014), “What Is Human Centred Design? “ The Design Journal, 17:4, pp. 606–623. ISO9241-210:2019: Ergonomics of human-system interaction — Part 210: Human-
centred design for interactive systems. ISO 26800:2011: Ergonomics - General approach, principles and concepts.
Conceptual Change in Human-centered Design by Artificial Intelligence System 151
Lewin, K., (1951), “Field theory in social science: selected theoretical papers,” Harper. OECD (2019), “Recommendation of the Council on Artificial Intelligence,” online
at: https://www.oecd.org/digital/forty-two-countries-adopt-new-oecd-principles -on-artificial-intelligence.htm Sunaga, T., et al. (2014), “Self-direction in a Co-design Project for a Hospital Information System: Toward Society-shaping Design,” INTERNATIONAL DESIGN CONFERENCE, pp. 2167–2176.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
The Human Side of Service Engineering
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-38-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional