Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Human-Centred Principles for the Design of Shape-Changing Tangible User Interfaces

School of Computer Science, University of Birmingham, UK ↗
Publication year
Digital Object Identifier10.54941/ahfe100865

Abstract

Shape changing interfaces (SCI) are interactive Tangible User Interfaces (TUI) that change materiality or shape as input and/or output (Alexander et al., 2018). While these represent novel technologies, there remain problems in understanding how to design, use or evaluate these. One reason behind such problems might be the lack of guidelines or frameworks that are developed specifically for SCI. Another reason can be the lack of theory of human performance to inform design decisions. In this paper, we review frameworks that have been proposed for SCI and relate these to a theory of affordance. While HCI has used the concept of affordance for many years (Norman, 1988), there is a tendency to assume that it relates solely to the physical properties of an object. To this end, it has been assumed that form (physical appearance of the object) permits function (a particular action with that object). However, the concept of affordance is much richer than this implies and takes into account the capabilities of the person performing the action, the goal that the person is seeking to achieve, the environment in which the interaction between person and object occurs, as well as the properties of the object. Further, as the object could be designed to have sensing and acting capabilities of its own, then there is a progression of interactions in which the object responds to person as much as the person responds to the object. To explore the concept of affordance, Baber (2018) developed Forms of Engagement (including environmental, morphological, motor, perceptual, cognitive, cultural). This reflects the ways in which people engage with artefacts and how different Forms of Engagement can serve to support and constrain each other. SCI design frameworks have focused on the type of shape change (Rasmussen et al., 2012; Roudaut et al., 2013; Kim et al., 2018) or purpose of different shapes (Alexander et al., 2018; Rasmussen et al., 2012). In this paper, the aim is to relate the type of shape change with the purpose of SCI. Such a framework will combine Forms of Engagement with properties and behaviours of SCI. This can simplify the application of affordance in SCI design, as it details different levels of affordance for each affording situation. Relating the shape change type with the function in a framework together with the affordance evaluation methodology would serve as a useful tool that will improve the usability of upcoming SCI. Moreover, this research will serve as a base for future studies that facilitate applying affordance concept to tangible user interfaces. The evaluation methodology can help SCI designers to assess their interfaces’ affordance or even provide the designer the ability to consider the criteria in the early stage of design to assist them create affording situations. ReferencesAlexander, J., Roudaut, A., Steimle, J., Hornbæk, K., Alonso, M. B., Follmer, S., & Merritt, T. (2018). Grand Challenges in Shape-Changing Interface Research Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal QC, Canada. https://doi.org/10.1145/3173574.3173873Baber, C. (2018). Designing Smart Objects to Support Affording Situations: Exploiting Affordance Through an Understanding of Forms of Engagement. Frontiers in Psychology, 9, 292. https://doi.org/10.3389/fpsyg.2018.00292 Kim, H., Coutrix, C., & Roudaut, A. (2018). Morphees+: Studying Everyday Reconfigurable Objects for the Design and Taxonomy of Reconfigurable UIs Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal QC, Canada. https://doi.org/10.1145/3173574.3174193Norman, D. A. (1988). The psychology of everyday things. Basic Books. Rasmussen, M. K., Pedersen, E. W., Petersen, M. G., & Hornbæk, K. (2012). Shape-changing interfaces: a review of the design space and open research questions Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, Texas, USA. https://doi.org/10.1145/2207676.2207781Roudaut, A., Karnik, A., Löchtefeld, M., & Subramanian, S. (2013). Morphees: toward high "shape resolution" in self-actuated flexible mobile devices Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France. https://doi.org/10.1145/2470654.2470738

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

INTRODUCTION

Shape-changing interfaces (SCIs) are interactive tangible user interfaces (TUIs) that change materiality or shape as input and/or output [1]. While these represent novel technologies, there remain problems in understanding how to design, use, or evaluate them. One reason behind such problems might be the lack of guidelines or frameworks that are developed specifically for SCIs. Another reason could be the lack of a theory of human performance to inform design decisions. SCI design frameworks have focused on the type of shape change [2-5] or the purpose of different shapes [1, 3, 6]. However, these individual efforts have not been unified in one SCI framework to facilitate the SCI design process. In this paper, we describe the development of an SCI design framework represented in an equation that relates the shape change type with the function together with an affordance theory. We believed that the SCI equation would serve as a useful tool that would improve the usability of upcoming SCIs.

SCI EQUATION We introduce an SCI equation that states the following:

A shape-changing interface, x, is an artifact in an application domain that fulfils a purpose by responding to input and employing types of shape-change through using an actuation system and modifying kinetic properties to support forms of engagement.

The intention behind the SCI equation is to provide a pattern or template against which the SCI designer can consider options. Ultimately, this could be supported by a software tool to present examples of combinations of the italicised elements. The main elements of the SCI equation are summarised in Figure 1. In the following sections, we provide explanations of each element.

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Figure 1: The elements of the SCI equation
Original PDF Figure 1:

Application Domain The application domain specifies where the SCI will be used. This will include both the physical environment, such as the home or office, and the general area of activity, such as entertainment, medical, and augmented living. Each application domain will specify requirements that need to be considered.

Purpose Purpose specifies the artifact function or goal. The artifact may change its shape to notify the user with either urgent or normal notifications. The responsive bracelet developed by Lu et al. (2019) has two buttons that inflate when the user receives a call; thus, the change in the buttons’ shape serves as a notification. Moreover, these inflated buttons encourage the user to press one of them in response to the incoming call. This possibility for action presented in the inflated buttons serves another functional purpose of the shape-changing feature, which can be considered dynamic affordance. Another example of a shape-changing artifact that provides dynamic affordance is SociaBowl by Leong et al. (2019). SociaBowl is a dynamic table centrepiece that might influence interactions between people around the table. The SociaBowl could be a container of pens that nods to a particular person at the table to encourage him to participate. Follmer et al. (2013) discussed how a shape change can be used to mediate interaction, including facilitation, refusal, or manipulation. Davis (2020) extended the possible mediated interactions to include requesting, demanding, allowing, encouraging, discouraging, or refusing an action.

Another purpose of SCI is to encode and communicate information to the user through combinations of visual, haptic, and shape animation. This is exemplified in a pin- based shape display called inFORCE, which can be utilised for volumetric data

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 representation in geoscience (Nakagaki et al., 2019). A SCI might communicate information in the form of synchronisation. reMi is a good illustration of synchronisation, as it is a tangible memory notebook that has shape-changing papers. The user can record the ambient sound in one of the papers and then replay the recorded sounds. The paper changes its shape, synchronising with the recorded sound (Choi et al., 2018). Physiological synchronisation, where the change in shape mimics organic and lifelike movement, is more popular in recent studies. The invisible physiological state of a person’s respiration patterns becomes tangible via a shape- changing interface called reSpire (Choi et al., 2019). Choi and Ishii (2020) developed a wrist-worn mobile heart rate rhythmic regulator named ambienBeat, which provides haptic feedback that synchronises with the user’s heartbeat. When the user’s heartbeat becomes too low, such as when the user is sleepy while driving, ambienBeat provides a fast frequency tactile stimulation to boost up the user’s heartbeat rate. Not only does ambienBeat change its shape for physiological synchronisation, but it also provides vibrotactile notification.

A shape-changing feature can also be used to overcome a problem. For instance, a robot wheel could transform its shape to a smaller size to move through small holes or adapt its shape to variable road conditions (Lee et al., 2021). A layer jamming technology has been introduced to design deformable furniture that provides portability (Ou et al., 2014). In soft robotics, the shape-changing feature has been used as a means of actuation, offering locomotion for the artifact. The 32-legged spherical robot, Mochibot, illustrates an example of a robot that changes its shape to accomplish smooth and continuous locomotion (Nozaki et al., 2018).

Researchers can develop SCIs as a product for the purpose of exploring human interaction with deformable artifacts over time (Zhong et al., 2021; Zhong et al., 2020). SCIs can assist in children’s learning process. Topobo and Kinematics are construction kits for children that have been designed as a playful and haptic way of learning about motion in the physical world. These construction kits not only serve as learning tools but also offer fun opportunities to the user.

The goal of a shape-changing feature in SCIs might not be for a functional but a hedonic purpose, such as aesthetics. Shape changing for aesthetics is demonstrated in the holes in the Enleon dress that open and close merely for decorative purposes (Berzowska & Mainstone, 2008).

Inputs Input specifies different possible inputs that an SCI might receive. Some SCIs generate output without receiving any type of input, and that occurs if the purpose of SCI is purely hedonic. Most SCIs receive inputs from direct or indirect human interactions with the artifact. Indirect interactions, such as proximity, generate sensor data that serve as an implicit input. Direct human interactions (e.g. touch) represent direct human inputs. It should be noted that an SCI might receive multimodal inputs; for instance, the user might touch the artifact by saying a word, making the inputs

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 consist of touch and voice (Ou et al., 2014).

Types of Shape Change The type of shape change specifies the aspects that will be transformed in the artifact. After reviewing the literature, such aspects of change can be categorised into six categories:

- Form: results in changing features, such as granularity, amplitude, zero- crossing, curvature, or closure (Roudaut et al., 2013)

- Size: results in changing the artifact’s length (1D), area (2D) and/or volume (3D) (Kim et al., 2018)

- Materiality: changing the artifact’s colour or texture that results in changing features such as stretchability, strength, and/or viscosity.

- Permeability: results in changes in the artifact’s porosity feature.

- Position: results in change in spatiality features and includes flappable artifacts.

- Composition: results in transformations that unite or divide elements of the artifact.

The choice of type of shape change depends largely on the purpose of the SCI. For notification purposes, changes in form, size, or composition have been found to be most suitable, whereas changes in form, size, materiality, or permeability are the most suitable for encoding or communicating information purposes (Pedersen et al., 2014; Rasmussen et al., 2016). Ishii and Parkes (2009) claimed that changes in composition are more convenient for SCIs designed for fun or learning purposes.

Actuation system The actuation system describes the system used to supply the interface with adequate energy to change its shape. There are different actuation types used in SCIs, including electromechanical, fluid-based, smart materials, and electromagnetic actuation.

Kinetic Properties Kinetic properties are physical specifications of the transformation that specify the way the previous types of change occur. The movement of the transformation in the form or size of the artifact, for example, might happen at different velocity (speed, acceleration, tempo, frequency), path (linear/curved, continuous/intermittent, smooth/jerky, pattern/random), direction (up/down, right/left, forward/backwards), or space (scale, form, kinesphere) (Rasmussen et al., 2012).

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Forms of Engagement Although human computer interaction (HCI) has used the concept of affordance for many years (Norman, 1988), there is a tendency to assume that it relates solely to the physical properties of an artifact. To this end, it has been assumed that form (physical appearance of the artifact) permits function (a particular action with that artifact). However, the concept of affordance is much richer than this implies and takes into account the capabilities of the user performing the action, the goal that the user is seeking to achieve, the environment in which the interaction between user and artifact occurs, and the properties of the artifact. Further, as the artifact could be designed to have sensing and acting capabilities of its own, there is, therefore, a progression of interactions in which the artifact responds to the user as much as the user responds to the artifact. To explore the concept of affordance, Baber (2018) developed forms of engagement including environmental, perceptual, cognitive, morphological, motor, and cultural. The concept of forms of engagement reflects the ways in which people interact with artifacts and how different forms of engagements can serve to support and constrain each other. However, these forms of engagement do not take kinetic properties into account. Thus, we propose a refinement of Baber’s forms of engagement by adding another form of engagement called affective engagement. Here is the intended meaning of each form of engagement:

- Environmental engagement: the ability to recognise salient features in an artifact

- Perceptual engagement: relates salient features to the changing state of the artifact-user system

- Cognitive engagement: provides high-level management on ongoing actions

- Morphological engagement: the effectivity of the user, such as hand size.

- Motor engagement: the ability to act using the artifact

- Cultural engagement: interpretation and response to an artifact’s form in terms of the user’s culture

- Affective engagement: users’ emotions, mood, feelings, and attitudes resulting in physical specifications of the artifact transformation.

Each of the forms of engagement contributes to all SCIs, but some of them might contribute more depending on the other elements in the SCI equation. To demonstrate how the SCI context of use has an influence on the SCI’s affordance, we can contemplate how a user perceives an SCI differently from one domain to another. For example, a chair that changes its height in an airport can make the user nervous or

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4 afraid, while the chair itself can be a source of enjoyment in a music concert (Grönvall et al., 2014). Regarding the type of shape change, research has proven that changing the form of the artifact contributes highly to the user’s morphological, motor, and cultural engagement in the artifact affordance, whereas changing the size contributes highly to their morphological, motor, and affective engagement (Cano & Roudaut, 2019; Follmer et al., 2013; Grönvall et al., 2014). The permeability change results in a high contribution to the user’s motor engagement with the SCI affordance, whereas changing materiality results in a high contribution to their motor, cognitive, cultural, and affective engagement (Cano & Roudaut, 2019; Follmer et al., 2013; Rasmussen et al., 2016). In terms of the actuation system, using a pneumatic actuator results in a higher contribution to the user’s motor engagement in the SCI affordance than using a mechanical actuator (Tiab & Hornbæk, 2016). The kinetic properties of an SCI have a direct influence on its affordance. This can be seen in the user perception of a continuous smooth movement of the artifact transformation as pleasant and peaceful (Rasmussen et al., 2012). Changing the shape of an SCI by modifying the velocity of the transformation results in a high contribution to the user’s motor engagement in the SCI affordance, whereas modifying the direction of the transformation results in a high contribution to their motor, cognitive, cultural, and perceptual engagement (Tiab & Hornbæk, 2016).

CONCLUSION

A framework that incorporates affordance theory could help SCI designers relate the shape-change type with the interface purpose and application domain, thus increasing the efficiency of TUI over GUI in some applications. We proposed an SCI equation that specifies a pattern or template against which the SCI designer can consider options. Ultimately, this could be supported by a software tool to present examples of combinations of the equation elements. Moreover, such a framework combines forms of user engagement with properties and behaviours of the SCI, which would simplify the application of affordance in SCI design by detailing different levels of affordance for each affording situation. Future studies might investigate the relationship between the different elements in the equation to develop a more comprehensive framework to help SCI designers choose the right input and output mechanisms to fulfil the SCI purpose in a specific domain.

REFERENCES

Baber, C. (2018). Designing Smart Objects to Support Affording Situations: Exploiting Affordance Through an Understanding of Forms of Engagement. Frontiers in Psychology, 9, 292. https://doi.org/10.3389/fpsyg.2018.00292 Berzowska, J., & Mainstone, D. (2008). SKORPIONS: kinetic electronic garments.

In ACM SIGGRAPH 2008 talks (pp. 92-94).

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Cano, C. F., & Roudaut, A. (2019). MorphBenches: Using mixed reality experimentation platforms to study dynamic affordances in shape-changing devices. International Journal of Human-Computer Studies, 132, 1-11. https://doi.org/https://doi.org/10.1016/j.ijhcs.2019.07.006 Choi, K. Y., & Ishii, H. (2020). ambienBeat: Wrist-worn Mobile Tactile

Biofeedback for Heart Rate Rhythmic Regulation Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction, Sydney NSW, Australia. https://doi.org/10.1145/3374920.3374938 Choi, K. Y., Shinsato, D., Zhang, S., Nakagaki, K., & Ishii, H. (2018). reMi: Translating Ambient Sounds of Moment into Tangible and Shareable Memories through Animated Paper The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings, Berlin, Germany. https://doi.org/10.1145/3266037.3266109 Choi, K. Y., Sumini, V., & Ishii, H. (2019). reSpire: Self-awareness and

Interpersonal Connectedness through Shape-changing Fabric Display Proceedings of the 2019 on Creativity and Cognition, San Diego, CA, USA. https://doi.org/10.1145/3325480.3329176 Davis, J. L. (2020). How Artifacts Afford : The Power and Politics of Everyday Things : The Power and Politics of Everyday Things. MIT Press. http://ebookcentral.proquest.com/lib/bham/detail.action?docID=6267308 Follmer, S., Leithinger, D., Olwal, A., Hogge, A., & Ishii, H. (2013). inFORM:

dynamic physical affordances and constraints through shape and object actuation Proceedings of the 26th annual ACM symposium on User interface software and technology, St. Andrews, Scotland, United Kingdom. https://doi.org/10.1145/2501988.2502032 Grönvall, E., Kinch, S., Petersen, M. G., & Rasmussen, M. K. (2014). Causing commotion with a shape-changing bench: experiencing shape-changing interfaces in use Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto, Ontario, Canada. https://doi.org/10.1145/2556288.2557360 Ishii, H., & Parkes, A. J. (2009). Phrases of the kinetic: dynamic physicality as a

dimension of the design process. Kim, H., Coutrix, C., & Roudaut, A. (2018). Morphees+: Studying Everyday Reconfigurable Objects for the Design and Taxonomy of Reconfigurable UIs Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal QC, Canada. https://doi.org/10.1145/3173574.3174193 Lee, D.-Y., Kim, J.-K., Sohn, C.-Y., Heo, J.-M., & Cho, K.-J. (2021). High–load

capacity origami transformable wheel. Science Robotics, 6(53). Leong, J., Wang, Y., Sayah, R., Pappa, S. R., Perteneder, F., & Ishii, H. (2019).

SociaBowl: A Dynamic Table Centerpiece to Mediate Group Conversations Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland Uk. https://doi.org/10.1145/3290607.3312775 Lu, Q., Ou, J., Wilbert, J., Haben, A., Mi, H., & Ishii, H. (2019). milliMorph -- Fluid-Driven Thin Film Shape-Change Materials for Interaction Design. https://doi.org/10.1145/3332165.3347956 Nakagaki, K., Fitzgerald, D., Ma, Z., Vink, L., Levine, D., & Ishii, H. (2019).

inFORCE: Bi-directional `Force' Shape Display for Haptic Interaction Proceedings of the Thirteenth International Conference on Tangible,

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Embedded, and Embodied Interaction, Tempe, Arizona, USA. https://doi.org/10.1145/3294109.3295621 Norman, D. A. (1988). The psychology of everyday things. Basic Books. Nozaki, H., Kujirai, Y., Niiyama, R., Kawahara, Y., Yonezawa, T., & Nakazawa, J.

(2018, 1-5 Oct. 2018). Continuous Shape Changing Locomotion of 32-legged Spherical Robot. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Ou, J., Yao, L., Tauber, D., Steimle, J., Niiyama, R., & Ishii, H. (2014). jamSheets: thin interfaces with tunable stiffness enabled by layer jamming Proceedings of the 8th International Conference on Tangible, Embedded and Embodied Interaction, Munich, Germany. https://doi.org/10.1145/2540930.2540971 Pedersen, E. W., Subramanian, S., & Hornbæk, K. (2014). Is my phone alive? a

large-scale study of shape change in handheld devices using videos Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto, Ontario, Canada. https://doi.org/10.1145/2556288.2557018 Rasmussen, M., Troiano, G., Petersen, M., Simonsen, J., & Hornbæk, K. (2016).

Sketching Shape-changing Interfaces: Exploring Vocabulary, Metaphors Use, and Affordances. https://doi.org/10.1145/2858036.2858183 Rasmussen, M. K., Pedersen, E. W., Petersen, M. G., & Hornbæk, K. (2012).

Shape-changing interfaces: a review of the design space and open research questions Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, Texas, USA. https://doi.org/10.1145/2207676.2207781 Roudaut, A., Karnik, A., Löchtefeld, M., & Subramanian, S. (2013). Morphees: toward high "shape resolution" in self-actuated flexible mobile devices Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France. https://doi.org/10.1145/2470654.2470738 Tiab, J., & Hornbæk, K. (2016). Understanding Affordance, System State, and

Feedback in Shape-Changing Buttons Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, California, USA. https://doi.org/10.1145/2858036.2858350 Zhong, C., Wakkary, R., Chen, A. Y. S., & Oogjes, D. (2021). deformTable: Crafting a Shape-changing Device for Creative Appropriations Over Time Designing Interactive Systems Conference 2021, Virtual Event, USA. https://doi.org/10.1145/3461778.3462112 Zhong, C., Wakkary, R., Zhang, X., & Chen, A. Y. S. (2020). transTexture Lamp:

Understanding Lived Experiences with Deformation Through a Materiality Lens Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. https://doi.org/10.1145/3313831.3376721

Human Interaction & Emerging Technologies (IHIET-AI 2022) Artificial Intelligence & Future Applications https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8989-4

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET-AI 2022): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8989-4

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →