Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Can Design Teachers Evaluate Students’ Products from an End-User Point-of-View?

Publication year
Digital Object Identifier10.54941/ahfe100311

Abstract

The purpose of design education is to teach future designers to create products that fulfil the needs, wishes and expectations of the targeted users. Therefore, it seems reasonable that teachers in design education should have knowledge on how users experience products and apply this in the evaluation of design assignments. The question is whether ‘teachers are able to estimate the user experience?’. To answer this question the correlation between the assessment of products done by users and by teachers is analysed, by assessing 76 products designed by students. The teachers assessment correlated strongly to the assessment done by a jury of end users, (ρ = 0.743, α < 0.000), if the products designed for general target groups (i.e. adults between 18 and 65 years of age without special disabilities or very specific problems and needs). However, no correlation was found between the assessment of teachers and a jury of end users of products designed for people with disabilities or very specific problems and needs (such as bed bound hospitalized children).

INTRODUCTION

In design education design students are trained to create products that fulfil the needs, wishes and expectations of the targeted end-users. During the education program students are coached and assessed mainly by teachers 1, sometimes assisted by designers and specialists from companies who are cooperating in the student design projects. It is important that the persons who train designer students have a clear understanding of the needs, wishes and expectations of the target groups and of methods to discover those in order to be able to teach students how to focus on end-users in their design work. A part of such education concerns the evaluation of products and services, 1 In Belgium, where this study was conducted, the Art Academies are no Universities yet (but will be integrated into the universities in 2015), so the teachers are not (yet) professors.

designed by students, with the end-user in mind. In order to do so teachers in design education should be able to understand users and to estimate how users experience products. Evaluation in education is widely studied, resulting in handbooks (e.g. Eger, 2010; Daams 2011; Dirken, 2006) and guidelines on the topic (for example Tuning EU, 2013). However, little is known about whether teachers evaluate products and services the same way as end-user do. A lot of research about evaluation in art and design is done, but it is mainly of a general nature. It describes whether evaluations are summative or formative, (Danvers, 2012) or it indicates that assessment should focus on the student, process and product (Harpe et al., 2009). Design assessment is even described as an artful practice that might be linked to a form of connoisseurship (Orr, 2010).

The goal of this study was to explore the relationship between assessments of design students’ products carried out by teachers on the one hand and by end-users on the other hand. The hypothesis in this research was: ‘teachers are capable of understanding the end-users, resulting in a positive correlation between the assessments of teachers and the assessments of end-users’. The purpose of design education is to educate designers who can design products which fulfil the needs, wishes and expectations of users. It can be assumed that user experience of a product is important to design education, and as a consequence user experience is supposed to be important in the assessment of products designed by students. Which means teachers should be able to understand users and to estimate how users experience products.

In this study the assessment of 76 student products executed by teachers and a jury of users was analyzed. The 76 products are divided into a group of products designed for general target groups (i.e. adults between 18 and 65 years of age without special disabilities or very specific problems and needs) and a group of products designed for people with disabilities or very specific problems and needs.

Although the hypothesis of this research is ‘teachers are capable of understanding the end-users’, it can also be assumed that designers and users do not share the same vision on products, especially when it comes to innovation. The long time between the introduction of innovative products and the acceptation of this product by the main potential users suggests that designers see products different than common people do. The time between introduction and acceptation can vary between several years to several decades, or even centuries (Rogers, 1995). For that reason, in addition to the research described above, a cognitive mapping was done with design teachers about the way users and designers look at products and assess them. The mapping can also reveal the design teachers view of users.

METHODS

Data Set In this research 76 products designed by students are studied. The products were designed by 54 students of the ‘Product Design Education’ program of the Media Art & Design-faculty (of the Limburg Catholic University College in Genk, Belgium). Some students made more than one product (see Table 1). The products were designed by students who were in either the second or third bachelor year (age of the students at the time of study is typically 18- 23 years) or in one of the master years (age at the time of study is typically 21- 26 years). Most products were made by bachelor students (90%) as shown in Table 2.

Table 1: Number of cases per designer
Original PDF Table 1:
Table 2: Case specifications: gender, study level, number of cases per designer Students’ gender
Original PDF Table 2:

Students’ Level of education

Depending on the assignment, each of the students had to create a working prototype of a product or a concept in form of a mock-up or non-working prototype. In addition, the students had to show how their product worked by a presentation (either in PowerPoint or Prezi). In Figure 1 an example of a concept and a working prototype is shown. Most of the products (56/76) are designed for a more general target group, for adults between 18 and 65 years of age without specific disabilities or very specific problems and needs and for a specific activity (for example photography). Twenty products in this study are products designed for people with specific disabilities or very specific problems and needs (for example a washbasin for hairdressers working with elderly) (see Table 3).

Table 3: type of assignment assignments, subgroup and expected end result Type of assignment end result number of cases
Original PDF Table 3:

Bicycle aid for teaching children to ride a bike working prototype 17 Camera support for making pictures at 2.5 m height concept model 10

Master assignments Free assignment; master graduation working prototype 5

Figure 1: left: an example of a concept of a new Hairdryer; right: an example of a prototype of a dog support for people in wheelchairs
Original PDF Figure 1:

Study Design Product assessment

Both design teachers and a jury of end-users, who were part of the target groups the students designed their products for, assessed the products. This assessment was done after the presentation of the product by the student. In order to ensure that the teachers and the users assessed the same characteristics of the designed products, both groups were asked to pay special attention to: 1) the functionality (i.e. ease of use, adjustability to the user, fulfillment of the users’ needs and wishes), 2) the design, (shape color, texture), and 3) the perceived maintenance of the products. A jury of teachers (3-5 teachers) and a jury of end users (3-10 members) anonymously and individually rated the products by giving a score between 1 and 20 (1=low; 20=high). The jury of users who performed the assessments were not involved in the design process. The teachers were specifically asked only to assess the product, not the design process.

Statistical Analyses The data were analyzed statistically, by means of Bivariate Correlations (α < 0.05). First the whole group was analyzed and afterwards the two subgroups separately.

Cognitive mapping In order to further understand the similarities and differences between the assessments of teachers and users and to understand the possible differences in the vision on products of designers and users, a cognitive mapping (Martin, 2012) was performed in a workshop at the 18th International Design Educational Meeting (IDEM), 2012. Ten design teachers, from different countries in Europe, South America and the Middle East, participated in this workshop. Two questions were addressed in the mapping: ‘Do designers assess products in the same way as users? Why or why not?’ The results of the cognitive mapping were summarized in the workshop and approved by the participants.

RESULTS

Comparison of the assessments of design teachers and end-users The scores for the students’ products given by teachers varied between 5 and 17 (scale 1-20), the average score was 11.96 (SD = 2.705). The scores given by the jury of end-users varied between 5 and 18, with an average score of 13.28 (SD = 2.789). Figure 2 shows the individual scores for each designed product.

Figure 2: assessment of the products by design teachers and users
Original PDF Figure 2:

The results of the analysis of the scores for all students’ products together showed a strong positive correlation between the assessments of design teachers and of end-users (ρ = 0.694 , α < 0.000; see Table 4). Analyses that were performed separately for each of the two subgroups of end-users (i.e. healthy adults and people with disabilities or very specific problems and needs) show different results. No correlation between teachers’ scores and users’ scores was found for products that were designed for people with disabilities or very specific problems and needs (ρ = 0.470, α = 0.066). For products that were designed for adults without a disabilities or very specific problems and needs, a strong correlation was again found (ρ = 0.743, α < 0.000).

Table 4: correlations between the assessment of users and teachers number of cases α ρ
Original PDF Table 4:

Designer teachers’ vision The results of the cognitive mapping were summarized in the cognitive mapping session workshop and approved by the participants. The main point of view of the designers teachers was: that the difference between designers and users is that users want products that are functional, user friendly, beautiful and familiar to them. New products need to have a relationship to products they already know. All these things are also important to designers, but designers see innovation important as well, much more then (most) users do. Designers (and design teachers) value form, function as well as innovation, while users value mainly form and function, innovation is less important to them. They (users) don’t want things that differ much from the products they are used to. Most people don’t want modern design or the latest new technologies. They prefer familiar products and products that resemble what they already know. Another conclusion of this workshop was: designing is also about communication. Users don’t speak the designers’ language, or rather designers don’t speak the users’ language. Designers don’t understand the users (completely). Because of their different approach towards products, designers find different aspects of the product important than the users do. It is difficult to estimate the user experience, especially for young beginning designers. It is only by experience and a lot of user involvement that designers are able to understand the user experience. Young designers often neglect this importance. Design education has the responsibility to increase the awareness of the design students and of the importance of user involvement in design.

DISCUSSION

The goal of this study was to explore the relationships between the assessment of products by teachers and by end- users. The hypothesis this research was: there is a significant positive correlation between the assessments of teachers and of end-users.

Correlations between teacher assessment and user assessment of products Although strong correlations, between the assessment done by teachers and the jury of end-users, were found in the analysis of total group of the students’ products, it is premature to conclude that teachers are always able to estimate the users experience. When products designed for the more general target group (healthy adults between 18 and 65 years of age, without special disabilities or very specific problems and needs) and products designed for people with disabilities or very specific problems and needs analyzed separately, different results were found. For products designed for the more general target group a positive correlation between the assessments by users and by teachers the strong correlation still stands. Apparently, for this subgroup teachers were able to estimate how users would experience the product. This was to be expected since user experience is an important focus of the education and the faculty and teachers pay a lot of attention to user experience, as mentioned in the introduction. This becomes clear in the many user-oriented courses in the curriculum of Product Design (such as ergonomics, psychology, emotional design, experience design, etc…, MAD-faculty, 2013). Moreover, user assessment of products designed by students are organized regularly during the curriculum (in several projects each academy year), enabling the teachers to keep their user knowledge up-to-date.

For products that were designed for people with a disability however, no correlations were found between the assessment of teachers and the jury of end users. A possible explanation for this is that it is simply impossible for a teacher to have in-depth knowledge of the needs of the large variety of specific users their students design for. The lack of correlation between the product assessments of the teachers and of the users emphasizes the importance of involving users in the assessment of products in design education. The goal of design education, as mentioned in the introduction, is to train designers that can create products that fulfil the needs, wishes and expectations of the targeted end-users. Because teachers are unable to fully understand all users it is necessary for student to be confronted with the users assessment of their products. It will make the designer students aware of the designers’ limited insight into users and the importance of user involvement in the design process. This is in line with the conclusion of the cognitive mapping with the design teachers were they emphasized the limited understanding of users of young designers and the importance for young designers to be aware of the importance and value of user involvement.

Differences in vision on products of designers teachers users The cognitive mapping showed that one of the difficulties in designing is communication (in broader sense). Designers don’t speak the users language (Kok et al, 2014). Daams (2012) and Dirken (2006) also stated that one of the problems in design is the mismatch between the product image of designers’ and users (the product image communicate different to designers than to users). The difference in the designers and users “language” results in products that may be logic and intuitive from the designers’ point of view, but not for users. Other researches also showed that designers and users interpret products differently. Van Kuijk (2009) concluded in his research about pre-use and post-use evaluations of electronic consumer products that there is a gap between expected and experienced usability, apparently users don’t interpret products the way designers do. Den Ouden (2006) concluded the same in her research: customers have certain expectations of the usability of products they buy. However, once customers use those products, many are not as user friendly as they appear. Products sometimes are so hard to use that consumers need assistance to use them, or even return or abandon the product.

Limits of this research This study was conducted in a design academy, the products that were assessed here concerned prototypes and concepts and as such often did not focus on secondary use like placing a buggy in the trunk of a car or replacing a battery. Therefore generalization of the findings in this paper should be done with care. It would be interesting to do a similar study on products that are already on the market, to see whether these correlations are still valid. This study was conducted in only one academy, to generalize the correlation between teacher an user assessment more research in other design schools and academies should be conducted, but this study does show the correlation for this specific situation, which is an indication that teacher and user assessment have similarities. Also, in the cognitive mapping in another gremium the relationship was affirmed.

The research itself has also some limitations. The assessments are done individually and anonymously on paper, but it is still possible that the users and teachers are influenced by each other because the products are presented to the users and the teachers at the same time in the same room and they were allowed to ask questions. The juries of end- users were rather small (3-10), especially for the group of products for people with disabilities or very specific problems and needs, nevertheless it gives an indication about how users would assess these products. To draw more general conclusions complementary research with al larger group of end-users should be conducted.

Although the cognitive mapping was done in an international group of designer teachers. Generalizing the designer teachers’ vision, about the differences of assessing products between users and designers (teachers), the formulated in the mapping should be done carefully, because the mapping was done in a small group of ten designer teachers. A cognitive mapping with a larger group of designers would be interesting to see whether this vision on the differences in the way people and designers look at products still lasts.

CONCLUSIONS

The results show that teachers seem to understand how users (adults between 18 and 65 years of age, without special disabilities or very specific problems and needs) evaluate products. Significant correlations were found between users’ evaluations and teachers’ evaluations. In the assessment of products for people with disabilities or very specific problems and needs no significant correlation was found between the assessments of users and the assessments of teachers. This may be caused by the fact that it is more difficult for teachers to estimate the user experience for such specific target groups. From this study, we conclude that user involvement in product assessment during design education is important, especially in the assessment of product for people with disabilities or very specific problems and needs.

ACKNOWLEDGMENTS

The authors would like to thank the students and teachers of Productdesign (MAD-faculty, Genk) and the organization of IDEM for their cooperation. The authors would also like to thank all the persons who did the user assessment.

REFERENCES

Daams B., (2011), “Productergonomie: ontwerpen voor nut, gebruik en beleving.” {product ergonomics: designing for usefulness, use and perception.} Undedisining, NL. Danvers, J, (2012), Qualitative rather than Quantitative: the assessment of arts education. Networks magazine, ADM-HEA,

website: http://www.adm.heacademy.ac.uk/resources/publications/networks-magazine/ Dirken, H. (2006) “Productergonomie: ontwerpen voor gebruikers.” {product ergonomics: designing for users.}, VSSD, Delft, NL,. Erger, A., (2010), Bonnema, M, Litters, E. , Voort, van der, M, Productontwerpen. {Product Design } Den Haag, Lemma, NL. Harpe de la., B. (2009), Peterson J.F, Frankham N., Zehner R., Neale D., Musgrave E., McDermott R., “Assessment focus in

studio: What is most prominent in architecture, art and design?”, International Journal of Art & Design Education, Vol.28, Nr 1, pp. 37-51(15). Kok, B.N.E., Slegers, K., Vink, P., 2014), Can Design Teachers evaluate Students’ Products from an End-User Point-of-View?, 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014? Submitted. Kuijk, J.I. van , (2009) “Managing product Usability: How companies deal with usability in the development of electronic

consumer products;” PhD Thesis, TU delft, faculty of industrial design engineering, NL MAD-faculty, (2013) “Curriculum Productdesign”, website: http://www.mad-fac.be/content/productdesign, BE. Martin, B., Hanington, B., (2012), “Universal methods of design”. Rockport publisher, USA. Orr S. (2010), Making Marks: assessment practises in art and design. Networks magazine, ADM-HEA, website: http://www.adm.heacademy.ac.uk/resources/features/making-marks-assessement-practices-in-art-and-design/ Ouden, Den, E. (2006) “Development of a design analysis model for consumer complaints: revealing a new class of quality

failure.” PhD Thesis, TU Eindhoven, NL. Rogers, E.M., (1995), “Diffusion of Innovations” 4th edition, The Free Press, New York, USA. Tuning 2013, “Approaches to teaching, learning and assessment in competences based degree programs”, Education and Culture DG, European community, EU.

Cover of the proceedings volume Advances in Social and Organizational Factors

Explore the complete digital volume

Advances in Social and Organizational Factors

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-4951-2102-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →