© 2022. Published by AHFE Open Access. All rights reserved. 75
76 Pereira
INTRODUCTION
User Experience (UX) is a multidisciplinary field that encompasses the entire journey of the user’s interaction with a company’s product or service, engi- neering, marketing, graphical, industrial, and interface design. The key to a high-quality user experience is to design the UX in a way that provides accessibility and usability for a variety of audiences. For this reason, the field and research processes of UX primarily focus on placing the human at the center of all design and developmental efforts. Many UX professionals in the field utilize design thinking that heavily involves user research methodologies such as persona making, card sorts, journey mapping, and more. These resea- rch methods pinpoint solutions to user-centered weaknesses and interactions with the business. Yet, many of these research processes assume the end-user is a person who is neurotypical because they essentially require the user to be physically, socially, emotionally, and intellectually capable.
The purpose of this project is to study implicit biases such as physical, cognitive, or cultural barriers to participation in UX research processes by surveying and interviewing UX professionals about their work in the field. The project analyzed the following research questions: to what extent do exi- sting UX Design research methodologies address and consider people with disabilities? How are people with disabilities either included or excluded in standard UX research methods? How are accessibility and usability truly being assessed using these UX research methods? How are UX professionals establishing inclusivity in their current work? This project will also further study the awareness of implicit bias within UX research methods among the UX community and will allow for conversation about the ethics of using these methodologies. Lastly, the UX field will benefit from this research because it will create more acknowledgment toward designing user experiences for marginalized groups that UX professionals ultimately hold responsibility for.
BARRIERS WITHIN UX RESEARCH METHODOLOGIES The UX field at its core exists because of the need to better understand a company’s audience or potential audiences. This need is meant to provide a seamless and accessible experience through the user’s complete journey of a company’s product or service. UX is built on the concept of the design cycle which consists of the five steps to empathize, define, ideate, prototype, and test. All these actions contribute to satisfying user needs and constant impro- vement of usability in the user journey (Dam and Siang, 2021). However, at each step of the user journey, user research can be implemented at any point of the design cycle (Farrell, 2017). Therefore, UX research methodologies are of utmost importance to the UX field as it is solely based on user feedback and findings from the research itself. Furthermore, to standardize user resea- rch within the UX field the International Standard was established by UXPA, an international UX committee, that places basic guidelines that define gene- ral processes for human-centered activities. However, the committee fails to specify the exact methods to be used to perform them (UXPA International, 2019). The UXPA also established an official Code of Conduct that places
Implicit Bias in UX Research Methods 77
ethical responsibilities among UX professionals. However, these principles do not address the presence of implicit biases in UX research methodologies.
Implicit bias in UX will be defined as the physical, social and emotional, or cognitive and intellectual barriers to participation in the UX research process. UX professionals hold a high level of responsibility in providing valid resea- rch methods regarding consciously avoiding implicit bias among vulnerable populations. With this pressure of responsibility, a study by Gray and colle- agues revealed that “dark patterns” exist in UX practices that disregard this power of authority because of organizational pressures and the prioritization of profitability of design above other social motivations (Gray et al., 2018). Also, educational curriculums that provide training in rigorous usability rese- arch methods in UX are limited (Barnum, 2019). While these researchers do not address the actual methodologies or tasks within user research that can contribute to unconscious or conscious implicit biases in user research, they highlight how workplace pressures and the educational backgrounds of UX professionals may create barriers to an inclusive user research process.
Articles and research on vulnerable populations in UX are often associ- ated with the dangers of ethical practices of integrating machine-learning artificial intelligence systems with UX user research (e.g. Yang et al., 2020; Bergstörm, 2021; Loi et al., 2019; Cramer and Kim, 2019). Although huma- nity as a whole and the rise of AI as a means of providing an inclusive UX experience is an important topic of discussion, there is little investigation on specific marginalized groups especially for people with disabilities. For exam- ple, concern over specific marginalized groups within UX research methods involve activities that require queering consent for the LGBTQ+, humanita- rian interventions for populations of poverty, a multilingual user experience for several, and a more inclusive experience for women of color in the UX design process (Rose et al., 2018). Creating an overall inclusive UX experi- ence for users with disabilities finds that there is little knowledge over how they interact with contemporary products and services. Additionally, there is an ongoing debate over whether accessibility guidelines fit the usability needs of users with disabilities that first begin in the UX research process (Oswal, 2019).
ANALYSIS AND EVAULATION OF IMPLICIT BIAS IN UX RESEARCH METHODOLGIES In the following I describe methods I used for the analysis of implicit bias in UX research methodologies and the results I derived from them.
METHODS
My research methodologies consisted of a mixed-method Qualtrics survey and an optional interview of UX professionals. A 15-minute Qualtrics survey and optional interview provided qualitative and quantitative data because the survey used Likert scale, open-ended, binary (yes/no), and multiple-choice (choose more than one or single-answer) questions. While the interview col- lected completely qualitative data that was used to expand on implications
78 Pereira
from the qualitative survey. The mixed-method survey consisted of 22 que- stions separated into three sections: 1) demographic data, 2) workplace data, and 3) implicit bias in UX research methodologies data. Firstly, demogra- phic data generated insight into the general, educational, and experiential background of the professionals in the field of UX. Secondly, workplace data entailed questions that involved the professionals’ use of UX research methodologies in their work and whether their workplaces make the effort to encourage inclusivity in their products and environment. Thirdly, impli- cit bias in UX research methodologies data involved questions that guided survey-takers to evaluate their current workflows using UX methodologies and the current state of UX methodologies in terms of implicit bias toward users of marginalized groups. Then at the end of the survey, the partici- pant was presented with a voluntary sign-up for a date through the Calendly platform to schedule a 30-minute interview through Zoom.
If the survey-taker volunteers for the interview, a time was set up and a consent form was required before the interview was conducted. The consent form was sent directly to the principal investigator and included informa- tion that ensures anonymity and an optional agreement to record an audio transcription of the entire session. To conduct the interview, one person took notes on a Miro board with the camera off, and another person assumed the role of the “emcee” that keeps the interview flowing in a concise and timely manner using the I-Teach method (Satterfield et al., 2021). The Miro board was revealed to the interviewee at the beginning of the session and then shut off until the end. Then the emcee asked qualitative questions that are divi- ded into three sections with major emphasis on 1) educational background, 2) implicit bias in UX job experience, and 3) an overall outlook to solutions to implicit bias in UX research methodologies. The Miro board was designed to organize responses in a timely manner, within the three sections, where some boards were used for open-ended phrased natural language responses and other boards were codified with sticky notes in single sentences. At the end of the session, there was a small section for questions and words of wisdom from the interviewee. Lastly, the emcee shared the screen of the Miro board with the filled-out responses and gave the interviewee a chance to correct anything that was inaccurate that was said on their behalf.
At the time of this writing, 25 respondents have fully participated in the research process, and 1 interview was conducted. The following results are found from the survey separated into three categories (Demographic Data, Workplace Data, and Implicit Bias in UX Research Methodologies Data).
RESULTS
Demographic Data Demographic data questions used to gain a better understanding of the range of careers within the UX field and to understand if respondents felt like their formal education has trained them well enough in ethical practices for UX. A majority of respondents hold a UX Design, Research, or UI/UX career role with over 44% having over 6+ years of experience and 56% having less than 5 years of experience. Over 96% of respondents have attended college
Implicit Bias in UX Research Methods 79
or higher education. However, most of the degrees achieved by respondents come from a wide variety of fields that are not directly tied to the UX field. From this demographic data was collected to find whether respondents felt like their formal education trained them well in ethical practices in UX. Over 80% of respondents had a positive experience with their formal education teaching ethical practices in UX, while 20% disagreed with this statement. This data overall evaluated how truly diverse the UX community is from a formal educational standpoint, and that there are many career paths that fall under the UX umbrella. It also acknowledges that ethical practice in UX is being taught at a sufficient level in formal educational institutions. Further- more, most UX professionals are highly educated. However, what is most important to note is that although these respondents had experienced a more positive education training in ethical practices in UX with their formal educa- tion, their responses about training about ethics in UX from their workplaces indicated less changes in this area.
Workplace Data Workplace data used questions to understand whether the workplaces of respondents are incorporating training in ethics and to find what specific UX research methodologies are being used in their work. 64% of respondents do not attain ethics workshops or training from the companies they work for. Furthermore, 66% of respondents did not have a guideline for incor- porating inclusivity in their workplace. Although most respondents had a positive formal educational upbringing toward ethical practices in UX, their workplaces did not carry the same goals toward reinforcing ethical guidelines or training. This information is critical because although formal education in ethical practice is principal, respondents’ companies failed to provide ethi- cal guidelines or training. These forms of ethical training or guidelines are paramount to delivering inclusive and accessible products to their audiences. Most respondents said that they consider accessibility in terms of catering to marginalized populations in projects and implementing diversity in their company’s hiring practices. This data reveals that inclusivity is mostly initi- ated by respondents’ selves rather than the work of the team or company. Especially by one response that claimed that “I am to increase the stakehol- ders’ understanding of their audience and fight for products to be built or modified to a11y standards.”
Lastly, respondents were asked what specific UX methodologies are used in their work. Out of 25 respondents, surveys and questionnaires were used by all of them. A close second were interviews at 23 respondents and usability tests at 22 respondents. Notably, journey mapping was used by 18 respon- dents, and persona making was used by 15 respondents. The commonality between all of these UX research methodologies is that they are most likely designed and conducted by a UX professional.
These research methodologies all involve physical, social and emotional, and cognitive or intellectual processes that may include possible implicit bias for marginalized groups. For example, implicit bias might be present in surveys and questionnaires that include its characteristics that involve
80 Pereira
language-based questions, length of time and number of questions, type-in or clickable interaction, and number/type of responses. Moreover, surveys can cause discomfort, presume an answer, cause a respondent to de-identify themselves, or present a null set that indicates a question that may be overap- plied or motivate triangulation in data analysis. For interviews, the implicit bias can be found in the deidentification of metadata in regard to body langu- age or visual presentation of the interviewee and power distance between the interviewer and interviewee. For usability tests, a user is interacting directly with the researcher where their physical, social and emotional, and cognitive or intellectual factors can place pressure or intimidation among the users. For journey mapping and persona making, these methodologies are most likely generative data by the researcher rather than being directly performed by a user. This UX method can be speculative or result in over-reading data or amplifying results in ways that could introduce biased assumptions about the user. All in all, there is a critical need to better understand implicit bias as it presents itself in surveys and questionnaires, surveys, usability tests, journey mapping, and persona making. Therefore, UX professionals need to under- stand how to mitigate implicit bias in each of these methodologies because of its prevalence in the UX industry.
Implicit Bias in UX Research Methodologies Data Implicit bias in UX research methodologies data directly asked participants of their awareness of the prevalence of implicit bias in UX research meth- odologies and the UX industry. The questions also gave respondents the opportunity to answer with a qualitative response. Any attempt to map the qualitative answers to the same response was based on an interpretation that I assigned it to. Difficulty to conduct UX research methodologies was asked between three vulnerable populations that included people with physi- cal disabilities, people with social and emotional disabilities, and people with cognitive or intellectual disabilities.
Firstly, participants were asked if they felt like it was difficult to con- duct UX research methodologies for people with physical disabilities. Of the responses, 10 out of 25 participants were on the agreed track, where par- ticipants responded that people with physical disabilities are a challenging population to recruit and accommodate. However, one respondent said that “it depends on the place you work for, but it’s also very dependent on how much effort the company/team is willing to accommodate those with physical disabilities. Essentially, you can make it work, but there’s a lot of red tape.” This places much importance on how accommodation is possible for this population, yet company and team dynamics play a large factor in choosing to accommodate people with physical disabilities with UX research metho- dologies. Additionally, 9 participants chose neither agree nor disagree, stating that remote research has made inclusive recruiting possible and that accom- modations are not difficult to make regardless of being a person who has a disability or other need. Furthermore, 6 participants chose the disagree track that indicated that conducting UX research methodologies for people with physical disabilities is completely possible and accessible where “as long as a
Implicit Bias in UX Research Methods 81
person can communicate, you can do the research”. Yet, there are limitations to what specific evaluation is needed for which UX research method to use among marginalized audiences.
Secondly, participants were asked if they felt like it was difficult to conduct UX research methodologies for people with social and emotional disabilities. Of the responses, 10 out of 24 participants who were on the agreed track stated that people with emotional and social disabilities are a challenging group to gain trust or robust data from using UX research methodologies. However, one participant communicated that “there are techniques that can get different abled individuals to express themselves through visual or other means”. Additionally, 9 participants chose neither agree nor disagree, sta- ting that it is not difficult to conduct UX research methodologies, yet it takes thoughtful writing and relevance to the study to this specific group. Furth- ermore, 5 participants chose the disagree track stating that self-patience and a key team of doctors or professionals will help with difficulties. Neverthe- less, honest data is an inevitable challenge with populations with social and emotional disabilities, yet UX research methodologies require aid or an addi- tional set of professionals to thoroughly conduct research without implicit bias.
Then participants were then asked if they felt like it was difficult to con- duct UX research methodologies for people with cognitive or intellectual disabilities. Of the responses, 13 out of 24 respondents were on the agreed track stating that UX methodologies make it difficult to create clear com- munication between the person with cognitive or intellectual disabilities. It is important to note that one respondent said that “UX research methodologies require lots of training to combat bias even for those without disabilities, so having a cognitive disability could make it much more difficult to conduct. Additionally, part of the UX research method while working with [the] audi- ence might require explanations that might be difficult for the researcher (if the disability is severe)”. In other words, UX methodologies require direct and quality forms of communication that limit the participation of people with cognitive or intellectual disabilities. Furthermore, 8 participants chose neither to agree nor disagree, reiterating that accommodations for people with disabilities should not be difficult as it is for other populations. Yet, 3 participants chose the disagree track, stating that every use case has its challenges and that “you have to get creative and rely on soft systems design teams to get through a subject”. Overall, verbal communication is difficult on the end of the person with cognitive or intellectual disabilities, there should be modes of training or accommodations that UX research methodologies should consider for.
Lastly, participants were asked if implicit bias exists in UX research methodologies. 20 out of 25 participants were on the agree track, while 4 participants were on the neither agree nor disagree track, and 1 participant remained on the somewhat disagree track. All in all, a majority of participants agreed that implicit bias exists in UX research methodologies. Although the sample was small, there is significant data that most participants had agreed to this statement that can later be used in a larger study.
82 Pereira
CONCLUSION
Trends in implicit bias were found to be prevalent within UX research meth- odologies from the results of this mixed-method pilot study. The survey data discovered that there may be insufficient training in ethics and guidelines for best practices of ethical UX research methodologies within workplaces where real-world products are developed and distributed. This idea contrasts with their positive experience of being trained from their formal educational insti- tutions about ethics in UX practices. Every day marginalized groups such as populations with physical, social and emotional, or cognitive and intellectual disabilities have been often ignored or seen as an afterthought in the utili- zation of these research processes. UX professionals hold the responsibility to provide experiences that are accessible among all humans and must con- tinuously examine ways to mitigate implicit bias. Especially since there is a high usage of surveys, interviews, usability tests, journey mapping, and per- sona making from participants, areas of implicit bias have been revealed to be commonly present. Also, accommodation and clear communication for populations with physical, social and emotional, or cognitive and intellectual disabilities need to be established within UX research methodologies.
Of the overall number of 65 participants, 25 continued to progress through the completion of the survey. Yet the survey data indicated the survey is usable and that most of the questions were answerable and valid. This is because most of the questions received over 25 responses by the total coh- ort of 25 participants in the study. Lastly, the most valuable questions were from the Implicit Bias in UX Research Methodologies Survey Data section. This section helped investigate specific UX methodologies being used by UX professionals and gathered the participant’s quantitative and qualitative data toward acknowledgment of implicit bias toward groups that have disabili- ties related to physical, social and emotional, or cognitive and intellectual abilities.
While the data for the interview was inconclusive, the strategy, rehearsal, and testing of the method was a valuable takeaway. Using two researchers with well-defined roles and a single participant, in addition to utilizing Zoom and Miro as data collection tools, created a smooth and fair environment for both the interviewer and the participant. Notably, vetting and validity of the interview data were accomplished by first allowing the participant to view the Miro board in the beginning to then shutting off the screen share of the Miro board until the end of the interview. Then resharing the Miro board to the participant to review and edit the Miro data at the end of the interview established validity by the participant themselves. All in all, the overall experience of conducting the methods was proven to hold a significant value that can be replicated for future studies that look at 100 to 1000s of participants with larger datasets.
ACKNOWLEDGMENT
I would like to first thank my thesis advisor, Debra Satterfield, who has men- tored and guided me through this research process. She is an intelligent and kind individual who has never hesitated to help me in regard to my career
Implicit Bias in UX Research Methods 83
aspirations, student projects, and life ambitions. I will always value her kno- wledge in the UX/Design field and her passion for inclusivity. Thank you for always being “on-call” and supportive throughout this journey.
I would also like to thank my professors in the UHP thesis process Dr. Caitlin Fouratt and Dr. Oliver Wang who have graciously taught me about research. They both have encouraged me and given me hope throughout this process. I thoroughly enjoyed connecting and learning from them.
Lastly, I would like to thank my family and friends who have always been by my side. Thank you for being in my life and always being so supportive of my life goals.
REFERENCES
Barnum, Carol. “Invited Essay: Usability Diverges, Media Converges, Design Remerges.” JUS, 2019, uxpajournal.org/usability-media-design/. Bergström, Emil. “Exploring User Experience Designers Experi- ences Working with Machine Learning.” DIVA, 4 June 2021, urn.kb.se/resolve?urn=urn%3Anbn%3Ase%3Ahh%3Adiva-44633. Cramer, Henriette, and Juho Kim. “Confronting the Tensions Where UX Meets AI.”
Interactions, vol. 26, no. 6, 2019, pp. 69–71., doi:10.1145/3364625. Dam, Rikke Friis, and Teo Yu Siang. “5 Stages in the Design Thin- king Process.” The Interaction Design Foundation, 2021, www.interaction- design.org/literature/article/5-stages-in-the-design-thinking-process. Farrell, Susan. “UX Research Cheat Sheet.” Nielsen Norman Group, 2017, www.nngroup.com/articles/ux-research-cheat-sheet/. Gray, Colin M., et al. “The Dark (Patterns) Side of UX Design.” Proceedings of
the 2018 CHI Conference on Human Factors in Computing Systems, 2018, doi:10.1145/3173574.3174108. International, UXPA. “About UX.” UXPA International, 21 Nov. 2019, uxpa.org/about-ux/. Loi, Daria, et al. “Co-Designing AI Futures.” Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion, 2019, doi:10.1145/3301019.3320000. Oswal, Sushil K. “Breaking the Exclusionary Boundary between User Experience and
Access.” Proceedings of the 37th ACM International Conference on the Design of Communication, 2019, doi:10.1145/3328020.3353957. Rose, Emma J., et al. “Social Justice in UX.” Proceedings of the 36th ACM International Conference on the Design of Communication, 2018, doi:10.1145/3233756.3233931. Satterfield, Debra, et al. “Evaluating Innovation Strategies in Online Education in Higher Education.” Advances in the Human Side of Service Engineering, 2021, pp. 219–225., https://doi.org/10.1007/978-3-030-80840-2_25. Yang, Qian, et al. “Re-Examining Whether, Why, and How Human-AI Interaction
Is Uniquely Difficult to Design.” Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, doi:10.1145/3313831.3376301.