Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Design Puzzibility: design idea exploration based on design puzzles with deep learning

1 Graduate School of Design, National Yunlin University of Science and Technology, Taiwan ↗
2 Department of Digital Media Design, National Yunlin University of Science and Technology, Taiwan ↗
Publication year
Digital Object Identifier10.54941/ahfe1004236

Abstract

The exploration of design ideas is a puzzle-solving process. The design puzzle as a framework motivates practitioners to explore more meaningful, refined, and preferable designs. This research tackles the factors that combine human-computer interaction on visual and semantic entities as hints to support the puzzles of concept design. Finding perfect references to express a specific design idea can be time-consuming and challenging. Mood boards have been used to motivate designers to visualize their creative concepts and convey insights to teams. This research illustrates the design puzzle with the mood board approach to support the idea exploration of web information system design. Several technical aspects of the design puzzle are surveyed and leveraged to achieve supporting design idea exploration with deep learning. With our computer-aided framework, designers may select the best combinations of design insights to work on further. To summarize, designers can use semantic and visual content analysis for problem-solving and idea exploration. This approach enables them to refine hinted ideas more flexibly and effectively.

INTRODUCTION

Exploring design ideas helps teams uncover potential themes and concepts, which can then be used to establish design objectives for a web system design. Researchers have found it similar to puzzle-making and solving (Chang, 2004) behavior to set expectations and define upfront scope and process based on the findings. By modeling these cognitive processes, researchers can identify problems to solve and better understand the techniques and methods that aid design puzzles as part of the idea exploration process. In general, design puzzles are based on user interaction and observation within a spe- cific context of design goals. Figure 1 presents an overview of the essential elements of the design puzzle.

During the insight exploration process in design, inconsistency and ambi- guity of spoken words can frustrate collaboration. Design specs would also

© 2023. Published by AHFE Open Access. All rights reserved. 196

Design Puzzibility: Design Idea Exploration Based on Design Puzzles 197

lack elaboration to carry on, causing its failure to gain traction. In practical terms, deep learning is a type of machine learning that involves using neural networks to identify and understand patterns within data. These networks can be taught to recognize and categorize visual and language-based features by analyzing datasets containing images, colors, and language patterns. By applying deep learning to design idea exploration, the generation of person- alized and diverse collections of stimuli that provide hints about a specific problem-solving concept can be leveraged. By exploring possible solutions with the advent of deep learning, design puzzles have the potential to be taken to the next level.

Figure 1: Relations between elements of the design puzzle (Chang, 2004).

Figure available in the original PDF paper.

Mood boards help convey complex ideas to teams and inspire designers to visualize their creative concepts. They have become a powerful tool for designers when visualizing ideas, themes, emotions, and aesthetics in a tangi- ble and accessible way. As an implementation of design puzzles, mood boards create personalized experiences that reflect the designers’ unique idea explo- ration process and preferences. To sum up, designers can delve into more complex problem-solving and idea exploration when they have design puz- zles backed by semantic and visual content analysis. This allows for hinted ideas to be refined more flexibly and impactfully. Our prototype is for a glanceable, comprehensive view of the design puzzle for idea exploration. The prototype tests how the ideation process can be developed to allow participants to create effective communication between team members. The interactive mood board approach can aid designers, and problem solvers in creating alternative visual ideas based on deep learning methods.

RELATED WORKS To explore the component framework around the Design puzzle, a couple of topics are reviewed. 1) Design puzzle as a computation paradigm. 2) Deep learning methods that are related to Design puzzles. A mood board tool is implemented to actualize and explain the framework for a variation and reference of the Design puzzle.

Design Puzzle as a Computation Paradigm Designers rely on personal and shared knowledge to connect their ideas and create a clear structure in a concept map. This approach can help orga- nize information effectively. At the early design stage, typical influences of structured differences between situated design principles exhibit an overall commonality as concept generation creativity techniques are used. In the

198 Chang and Chang

concept linking process, dynamic distributed knowledge and content can be linked (Lai and Chang, 2006).

Implementations to organize the information generated during the design process (Liao and Chang, 2014) in interactive systems that aim to visualize and support such design behaviors in advance with computation. Another suitable concept of Jigsaws (Lo et al., 2014) has addressed the common ele- ments to represent the functional features of the Design puzzle. These puzzle- like behaviors formalize divergent-convergent exploration into a structure shared between human designers and computer programs. Further interdis- ciplinary and cross-cultural research should be encouraged to co-create the next stage of design thinking. Common elements and features of the Design puzzle are listed in Table 1.

Table 1. Common elements and features of the design puzzle.

Table available in the original PDF paper.

Element Feature Information Hints Partially revealed design knowledge representations Visualization Puzzle goals How the design is satisfied or evaluated Prediction

Puzzle rules How the hints are operated or interacted with Topology How the outcomes are produced, manipulated Generation Puzzle-making Interactive process to control rules and hints cycle Multimodal Puzzle-solving Exploration process to create and evaluate the design Multimodal

Deep Learning Methods to Support Design Puzzle Machine learning is a process of mimicking (Cakmak et al., 2010) human intelligence and behavior patterns. It can be used to predict human decisions for a web information system and provide valuable answers that can help designers understand their target audience’s needs. To describe the desired functionality of this objective, several technical tasks should be surveyed and mapped with design puzzles. Each technique serves a different purpose in analyzing and interpreting visual content. By combining computer vision and natural language processing, these techniques comprehensively describe the content in an image. Generating image descriptions (Vinyals et al., 2017) has a Convolutional Neural Network (CNN) to extract image features. These features are then fed into a Recurrent Neural Network (RNN) trained to generate a sequence of words that describe the image. To create a sentence that accurately describes an image’s content, the RNN considers the context of the words that came before it in the series. It ensures that the resulting sentence is both grammatically correct and coherent.

Mood Boards as Motivation to Design Idea Exploration Mood boards have the potential to be an example of how aesthetic objects can connect senses and emotions, providing a link for people in creative industries and beyond. Research on the mood board approach (Endrissat et al., 2016) explores the coordination of independent actors and their sub-products using a visual mood board that maintains plurality while

Design Puzzibility: Design Idea Exploration Based on Design Puzzles 199

directing and aligning them. Research on affinity diagrams in the informa- tion industry is being studied as a valuable computational design resource for analyzing user behavior and emotions. (Lokman and Kamaruddin, 2010). It benefits the industry and academia in accessing users’ subjective experience with design. In contrast, as an example and reference of the design puzzle, the features are compared as listed in Table 2.

Table 2. Comparing the mood board and the design puzzle.

Table available in the original PDF paper.

Design puzzle Mood board Hints Images, text descriptions, user tags Puzzle goals Exploring insights or patterns for creating new designs Puzzle rules Clustering images, semantic analysis, tagging, segmenting ideas Puzzle-making Collecting images, sorting by affinity Puzzle-solving Analyzing semantic and visual information, finding patterns

LEVELS OF DESIGN PUZZIBILITY Several topics for each level are considered from the ground up to build the suitable knowledge levels required for actualizing the Design puzzle. Figure 2 shows the overall levels of knowledge of the Design puzzle to be developed.

Figure 2: Implementation component framework around the design puzzle.
Original PDF Figure 2:

The proposed knowledge levels are listed as follows:

1) The components and relations are the foundation combined with the rules and hints of the puzzle and the goals that the user operates the Design puzzle to solve. 2) The behavior flows between different components, such as visualizing the hints to the user, the interactive process with the rules, how the rules modify the goals, and how the goals reveal hints. 3) The computation tasks that are built to evaluate the results produced during the process of puzzle-making and puzzle-solving. The result gen- eration process is controlled by the rules with user interaction. The hints extraction process from the generated results to visualize for the user to explore.

200 Chang and Chang

4) The deep learning model or otherwise processes to adapt the model with the data produced from puzzle-making and puzzle-solving tasks to pre- dict, recognize, or categorize the idea entities, the control policy, and the generated results of the design puzzle. 5) The context level in which the puzzle-making and puzzle-solving are to be applied as the design exploration process.

IMPLEMENTATION Framework: The Components and Relations

To implement the base of the Design puzzle, Figure 3 presents components such as the User, Hints, Rules, and Goals as stages of the puzzle-making and solving process. Components such as the Result, Policy, and Idea Entities are parts based on machine learning processes: 1) Result is content generated with the control of Rules. It can be used to extract Hints for the User’s further behaviors. 2) Policy is the feedback loop from the evaluation by Goals to actively feedback and modify for new Rules to generate new Results. 3) Idea Entities are new pieces of knowledge extracted from the evaluation by Goals to provide updates to Hints. Lastly, Context is the application level for tasks of different variations of design puzzles. For instance, the mood board tool is the specific context of this research.

Figure 3: Component framework of the design puzzle implementation.
Original PDF Figure 3:

Context: The Interactive Mood Board Tool Considering the essential implementation is primarily on adapting knowledge from data that users produced within its application, we limited the environ- ment held on Miro (https://miro.com), a digitized real-time whiteboarding tool accessible globally directly for users. The notes and groups are recorded and aggregated as a CSV format data file, with fields as Table 3 shows.

Design Puzzibility: Design Idea Exploration Based on Design Puzzles 201

Table 3. Comparing the mood board and the design puzzle.

Table available in the original PDF paper.

Field Name Data Type Description Image Bitmap Bitmap file of the image data Title Text Title of the image Description Text Description text about the image UserTags Array Group of words about the image

With the data fields of workshop-generated content, the learning process tends to adapt knowledge from the user tags and clusters of images, which are crucial for a mood board tool for deciding further objectives and goals to design.

Figure 4: The mood board tool with unsupervised machine learning mechanisms.
Original PDF Figure 4:

Integration: Components Supported by Deep Learning The exploration mechanism has an interactive process consisting of three main steps: 1) Behavior, 2) Interpretation, and 3) Representation. Until a satisfying result is reached, users are allowed to add new notes or edit existing notes in each step; the refining loop of the exploration process is rendered in Figure 5.

Figure 5: Design puzzle data processing and inferences with the mood board tool.
Original PDF Figure 5:

202 Chang and Chang

The initial stage is to emulate the behavior and process of design ideation. Various technical tasks are assessed and utilized to facilitate the exploration of design ideas through deep learning: 1) Image feature extraction and clus- tering. 2) Image-to-text captioning and description. 3) semantic synthesis and text generation with pre-trained language models.

Figure 6: Design puzzle data processing and inferences with the mood board tool.
Original PDF Figure 6:

ANALYSIS AND EVALUATION The first part of the system, namely Moozzle, involves collecting images and embedding information for entity recognition of latent semantics. The second part of the automation clusters the images as knowledge graph-like mood board placement, which can present the visualization and prompt text gener- ated with image captioning. The third part of the process provides intuitive interaction methods and supports participants to explore and evaluate during a remote mood board session.

A Digital Mood Board Session During the workshop, the below steps are followed with the prototype Moozzle:

(1) When the session begins, participants are guided to refer to a manual mood board session to collect images for the concepts and collage the images. (2) Participants are asked to cluster the images with the similarity of their ideas about the image. Then, Moozzle scans through the images chosen by each participant and presents a phrase or sentence per cluster. (3) Moozzle infers which cluster text and suggests a movement arrow for the images toward the clusters where it is possible to belong to a cluster or sub-category under the existing cluster. (4) During the session, participants view, move around, and rearrange the images to fit the similarity relation between clusters.

Design Puzzibility: Design Idea Exploration Based on Design Puzzles 203

(5) Moozzle continues to prompt a recommended label when a cluster of images is closer to its latent semantic similarity that reflects the idea the cluster tends to address. (6) Finally, participants organize and evaluate to score how vital the groups and notes are. Moozzle prompts a recommended score on each cluster of images. Depending on the number of images and the similarity between the clustered images, the design objectives are presented as a result for participant’s decision for the further design objectives.

Feature Evaluation An observation assessment of the features of Moozzle’s deep learning- supported process and outcome has been derived. As listed in Table 4, Moozzle provides neither strong suggestions nor explicit directions for par- ticipants. However, it functions as a recommendation giver based on the patterns learned from the interaction-produced process. The knowledge of insight exploration can be extracted through the participants’ performance during the mood board session.

Table 4. Observation assessment of supported and non-supported works.

Table available in the original PDF paper.

Session activity Mood board with deep learning models Pre-workshop guidance Human only Participants collecting content Human only, Free of restriction Content inference and treatment Image feature similarity clustering + Human Scoring contents Semantic similarity + Human Clustering and labelling Image captioning + Human Viewing and rearranging contents Human behavior only Participants communication Unexpected stimulation by human interaction Identifying design objectives Extra recommendations by human interaction

CONCLUSION

As a typical variation of design puzzles, mood boards are utilized to create personalized experiences that reflect the designers’ unique idea exploration process and preferences. Mood boards are a helpful tool for designers to communicate complex ideas to their teams and to help them visualize their creative concepts. They have become a powerful way to represent ideas, themes, emotions, and aesthetics in a tangible and accessible way. Using an interactive mood board approach can assist designers and problem solvers generate diverse visual ideas that foster creativity through deep learning and a range of parameters.

Several lessons are learned:

  • User interaction and visualization of the Design puzzle - Provide both cog- nitive level and computational mechanism of design as a puzzle-making and solving process.
  • Motivation is essential for guiding the interactive behaviors needed for the puzzle exploration process.

204 Chang and Chang

  • Motivation for solving a puzzle should provide the mechanism for the components of design puzzles.
  • Explore alternative solutions - Solving puzzles requires a clear and deter- ministic goal and also permits the possibility for creative and alternative searches (Akin and Akin, 1998).

In general, the mood board is a variation of the design puzzles supported by semantic and visual content analysis, and designers are empowered with more complex problem-solving idea exploration in which hinted ideas to be refined effectively and accurately.

REFERENCES

Akin, Ö. and Akin, C. (1998), “On the process of creativity in puzzles, inventions, and designs”, Automation in Construction, Vol. 7, No. 2, pp. 123–138, doi: 10.1016/S0926-5805(97)00057-5. Cakmak, M., DePalma, N., Arriaga, R. I. and Thomaz, A. L. (2010), “Exploit-

ing social partners in robot learning”, Autonomous Robots, Vol. 29, No. 3, pp. 309–329, doi: 10.1007/s10514-010-9197-9. Chang, T.-W. (2004), “Supporting Design Learning with Design Puzzles”, in Van Leeuwen, J. P. and Timmermans, H. J. P. (Eds.), Recent Advances in Design and Decision Support Systems in Architecture and Urban Planning, Springer Netherlands, Dordrecht, pp. 293–307, doi: 10.1007/1-4020-2409-6_19. Endrissat, N., Islam, G. and Noppeney, C. (2016), “Visual organizing: Balanc-

ing coordination and creative freedom via mood boards”, Journal of Business Research, Vol. 69 No. 7, pp. 2353–2362, doi: 10.1016/j.jbusres.2015.10.004. Lai, I.-C. and Chang, T.-W. (2006), “A distributed linking system for supporting idea association during the conceptual design stage”, Design Studies, Vol. 27, No. 6, pp. 685–710, doi: 10.1016/j.destud.2006.03.002. Liao, M. N. and Chang, T. W. (2014), “EVA: A Visualization Framework for

Organizing Concept Sketches”, 2014 18th International Conference on Infor- mation Visualisation, presented at the 2014 18th International Conference on Information Visualisation, pp. 398–401, doi: 10.1109/IV.2014.57. Lo, C.-H. N., Chang, T.-W. and Lai, I.-C. (2014), “Assembling the unexpected inspiration–from linking to jigsaw”, Frontiers of Architectural Research, Vol. 3, No. 4, pp. 358–367, doi: 10.1016/j.foar.2014.04.005. Lokman, A. M. and Kamaruddin, K. A. (2010), “Kansei affinity cluster for affective

product design”, 2010 International Conference on User Science and Engineering (i-USEr), presented at the 2010 International Conference on User Science and Engineering (i-USEr), pp. 38–43, doi: 10.1109/IUSER.2010.5716719. Vinyals, O., Toshev, A., Bengio, S. and Erhan, D. (2017), “Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge”, IEEE Transac- tions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 4, pp. 652–663, doi: 10.1109/TPAMI.2016.2587640.

Cover of the proceedings volume Human-Centered Design and User Experience

Explore the complete digital volume

Human-Centered Design and User Experience

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-90-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →