Toby Lu (2024). BlindSpot: An AI-Powered Intelligent Mirror Assisting with Facial Hygiene Analysis for Blind People. In: Human Interaction and Emerging Technologies (IHIET 2024). AHFE Open Access, vol. 157, pp. 271–278. https://doi.org/10.54941/ahfe1005486
Appearances are important for our everyday lives: it is how we present ourselves. Yet, for people who are blind or visually impaired, it can be challenging to notice or take care of their facial hygiene without sighted assistance. Furthermore, existing AI-powered visual assistive technologies such as Seeing AI do not provide such support.We first performed an online search to identify the needs of blind people and people who are visually impaired by checking their appearances. We found that checking appearances is a pertinent need shared by many people who are blind or visually impaired. For example, in the autoethnography by a blind girl [1], the author explains that growing up with visual impairments is physically and mentally strenuous on teenagers. In Li et al. [2], the authors find that people who are visually impaired “normally keep [their] appearance the same as before the loss of vision. This makes people focus less on [their] face and [their] visual impairments.” And that “people with visual impairments are attentive to their appearance,” by “applying makeup”, so that they “can control [their] appearance again.” Pradhan et al. [3] found that people with visual impairments care about their appearance as “they know people around them can see them,” they also “rely on sighted or partially sighted people to act as their mirrors.” Such prior work highlights the need to develop novel solutions and mechanisms to help blind people check on their appearances more independently. The advances in artificial intelligence, especially with regard to AI’s capabilities on analyzing images, brings tremendous potential to augment the vision of people who are blind or visually impaired. In this project, I propose the design of BlindSpot, which is an AI-powered intelligent mirror to help people who are blind and visually impaired check their appearances through uploading images to the platform. Our prior need-finding search revealed diverse needs by blind people on checking appearances, including checking food stains around mouth area, checking makeup quality, and checking cleanliness of their faces. To use BlindSpot, we imagine users can first upload a reference photo which displays their regular appearance, and then upload another photo that they wanted to have the intelligent mirror check for them. In cases where users do not have a reference photo, they can directly upload a photo of interest and ask BlindSpot to check their appearance for them. To evaluate the design concept of BlindSpot, we first constructed a repository composed of people’s headshot images with different appearance issues ranging from food stuck on teeth, to smudge of makeup. We then prompted GPT-4 to evaluate the headshot images and give users feedback for them to adjust their appearance. We found that in most of the scenarios, GPT-4 is able to generate accurate assessment and give helpful feedback. We report scenarios where GPT-4 makes mistakes and makes suggestions on how to further improve the design.
Appearances are important for our everyday lives: it is how we present ourselves. Yet, for people who are blind or visually impaired, it can be challenging to notice or take care of their facial hygiene without sighted assistance. Furthermore, existing AI-powered visual assistive technologies such as Seeing AI do not provide such support. We first performed an online search to identify the needs of blind people and people who are visually impaired by checking their appearances. We found that checking appearances is a pertinent need shared by many people who are blind or visually impaired. For example, in the autoethnography by a blind girl (Holly, 2019), the author explains that growing up with visual impairments is physically and mentally strenuous on teenagers. In Li et al. (2022), the authors find that people who are visually impaired “normally keep [their] appearance the same as before the loss of vision. This makes people focus less on [their] face and [their] visual impairments.” And that “people with visual impairments are attentive to their appearance,” by “applying makeup”, so that they “can control [their]
appearance again.”Pradhan and Daniels (2021) found that people with visual impairments care about their appearance as “they know people around them can see them,” they also “rely on sighted or partially sighted people to act as their mirrors. ”
Table 1. An overview of how BlindSpot works. The user may choose to upload a reference photo as the baseline (left photo). When the user has a need to check their appearances, they can upload a user taken photo (right photo). BlindSpot will give them feedback as shown.
Table available in the original PDF paper.
User reference photo User taken photo AI feedback
[1]
The photo you’ve provided appears to be a stylized and polished look, typically seen in makeup artistry for special events or professional photography. Everything seems to be intentionally crafted, and there doesn’t appear to be anything that needs cleaning up in terms of makeup application or hair styling from what I can discern. Each element from the hair to the makeup is designed to create a glamorous and cohesive appearance.
Such prior work highlights the need to develop novel solutions and mechanisms to help blind people check on their appearances more independently. The advances in artificial intelligence, especially with regard to AI’s capabilities on analyzing images, brings tremendous potential to augment the vision of people who are blind or visually impaired. In this project, I propose the design of BlindSpot, which is an AI-powered intelligent mirror to help people who are blind and visually impaired check their appearances through uploading images to the platform.
Our prior need-finding search revealed diverse needs by blind people on checking appearances, including checking food stains around mouth area, checking makeup quality, and checking cleanliness of their faces. To use BlindSpot, we imagine users can first upload a reference photo which displays their regular appearance, and then upload another photo that they wanted to have the intelligent mirror check for them. In cases where users do not have a reference photo, they can directly upload a photo of interest and ask BlindSpot to check their appearance for them.
To evaluate the design concept of BlindSpot, we first constructed a repository composed of people’s headshot images with different appearance issues ranging from food stuck on teeth, to smudge of makeup. We then prompted GPT-4 to evaluate the headshot images and give users feedback for them to adjust their appearance. We found that in most of the scenarios, GPT-4 is able to generate accurate assessment and give helpful feedback. We report scenarios where GPT-4 makes mistakes and makes suggestions on how to further improve the design.
BlindSpot: An AI-Powered Intelligent Mirror Assisting With Facial Hygiene Analysis 273
RELATED WORK
In this section, we will synthesize prior work that has developed assistive technologies for blind or visually impaired people (BVI). We will first introduce the challenges BVI people experience in their daily routines as shown by prior work. Prior research papers have described the challenges people that are visually impaired experienced in daily lives. For example, Huber et al. (2022) did a multi-year study to explore how visually impaired people can be better supported in their indoor navigation. Moreover, Yairi et al. (2011) proposed a novel graphic representation method with touch and sound as the universal designed touch-screen interface for visually impaired people to understand graphical information. In another study Yairi et al. (2009) developed a method to help visually impaired people intuitively recognize maps using audio and touch panels. This method is universal- designed to help both visually impaired and non-impaired people to explore maps together.
Prior work has also described BVI people’s challenges with regard to independent living and everyday activities. For example, prior study by Li et al. (2022) identified a variety of daily living tasks people with upper-body motor impairments encounter. Moreover, Fusseneger and Spiel (2022) used an auto-ethnographic approach to understand the function and relevance of assistive technologies in everyday life. They found that assistive technologies may pose a risk to users, especially when there is a technology failure.
NEED FINDING To understand BVI people’s needs with regard to checking appearances in their daily routines, we first performed a need finding search through online resources to identify the circumstances BVI people would like to check their appearances. We initiated the search through prompting ChatGPT specifically for links to blog posts written by BVI people. Here is a summary of findings.
Some blogs posted by people with visual impairment described their personal experience as “emotional and physically strenuous” Holly (2019). Some other articles have shared quotes from blind people describing their experiences Mohapatra (2018), for example, “I normally keep my appearance the same as before the loss of vision. This makes people focus less on my face and my visual impairments.” “Nonetheless, people with visual impairments are attentive to their appearance in the same proportion as sighted peers, especially in spaces that are guided by social norms around makeup.” “When I first lost my eyesight, I was quite sad that I couldn’t look in the mirror. Applying makeup is a way that I can control my appearance again.”
Specific to checking appearances, prior work has shown that “Blind individuals pay a lot of attention to their appearance because they know people around them can see them” Pradhan and Daniels (2021). It’s also disclosed in this study that “blind people rely on sighted or partially sighted people to act as their mirrors.”
It is shown from prior blog posts and studies that blind people do care about their appearances, e.g., “She obviously cared about her physical
274 Lu
appearance, as I noticed from her expertly applied make-up and glorious hair” (Quora, 2018). “Blind people are concerned with things about as much as the sighted people around them are, such as their readers, their companions, their coworkers, their friends and families” (Quora, 2018). Posts from Reddit also show that blind people care about their appearances, e.g., “Yes, because they want to fit in like anyone else.”“They still want to be presentable and put-together” (Reddit, 2019).
A YouTube (YouTube, n.d.) video shows that a blind girl enjoys doing makeups for herself and become youtubers guiding other BVI people to do makeups, e.g., “HEY! I’m Molly, a typical sushi and makeup loving millennial girl who just so happens to be blind! I was diagnosed with Retinitis Pigmentosa at just 4 years old and began public speaking at age 5. Even though I can’t see, I know that there are bright spots in everything we face. Let’s find them together!”
As a summary, blind people care about their appearance. Their concerns may be based on factors like personal pride, comfort, cultural influences, and social acceptance. Blind individuals often use their other senses, such as touch and sound, to assess and enhance their appearance. Like anyone else, attitudes toward appearance vary among individuals (HousingLin, n.d.; Tindall, 2019).
USER STORY Jack will be attending an office party and he wants to check his appearance before going to the party. Since he had just eaten dinner and woken up from a nap, he wanted to check whether there was food stuck on his teeth or whether his hair was presentable. He opened BlindSpot and took a selfie of his whole face and another selfie of him smiling, showing his teeth. Using the voiceover from his phone, he asked BlindSpot to check his teeth and appearance. BlindSpot gave him feedback that the selfie was incomplete, and the right side of his face is cut off in the picture. Jack then retook a photo and asked BlindSpot to check appearance for him. This time, BlindSpot told him that there’s spinach on his right lateral incisor, so Jack cleaned it up. Jack then felt more confident to attend the party.
EXPERIMENTS I performed preliminary experiments to demonstrate the feasibility of the approach.
First of all, I compared providing a reference photo versus not providing a reference photo for ChatGPT to give feedback based on an image. I found that providing a reference photo made the AI give much more comprehensive feedback to the user, as shown in Table 1.
Next, I’ll present some preliminary results on the mistakes AI may make in this scenario. In my experiments, I found that ChatGPT did a reasonable job detecting anomalies on the person’s face. For example, in one photo where the person has smudged makeup, ChatGPT was able to successfully determine the smudge of lipstick on the person’s lower lip on their right side. However, they are unable to determine the smudged eyeliner, this may be due to the
BlindSpot: An AI-Powered Intelligent Mirror Assisting With Facial Hygiene Analysis 275
ambiguity of the aesthetic the AI thinks the person is going for. Moreover, when the person as noticeable dirt on their face, ChatGPT is able to capture them. The results are shown in Table 2.
Table 2. Some example outputs from ChatGPT given a user-provided photo. The ChatGPT generated feedback is shown in the middle column, and some of my interpretation is shown in the right column.
Table available in the original PDF paper.
Photo ChatGPT Feedback Notes
[1]
ChatGPT was able to successfully determine the smudge of lipstick on the person’s lower lip on their right side. However, it does not detect the smudged eyeliner, this may be due to the ambiguity of the aesthetic the AI thinks the person is going for; where the smudged lipstick is definite.
In the photo you’ve provided, you appear to have a smudge of lipstick on your lower lip extending slightly outside the natural lip line on your right side, it’s a minor detail, but touching up that area could create a cleaner look. Other than that, your appearance seems quite purposeful and expressive, suggesting a potential creative or artistic context for the photo.
ChatGPT was able to successfully determine smudges or spots of dirt scattered across the person’s face, as well as the different sides the dirt was on. It is important to note that the dirt spots are very noticeable.
[2]
In the photo you provided, there are noticeable smudges or spots of dirt scattered across your face, primarily on your cheeks, nose, and forehead. Here are the details regarding your appearance based on your left and right sides ….
[3]
ChatGPT at first, was unable to detect anything odd with the person’s teeth, however, when asked specifically about the teeth area, they were able to notice the piece of green material on the person’s teeth. ChatGPT was also able to correctly identify the area where the material was.
Upon reviewing the image you’ve provided, it appears that there is a small, dark green piece of what could be food or another material on the tooth that is second from the left on the top row of the teeth (from the viewer’s perspective). This is on your right side. It’s subtle, but if this were for a professional setting or a photo you intend to share, you might want to clean it off.
I also noticed that when giving specific prompts, ChatGPT is more likely to detect things that are aligned with the users’ expectations. For example, when using a generic prompt “Do you notice anything in this photo of me? Is there anything wrong and/or noticeable with my appearance? Is there anything I
276 Lu
need to clean up? Also make sure you tell me these things regarding MY lefts and rights.”, ChatGPT sometimes can’t detect food stuck on someone’s teeth, however, when the prompt is specifically designed to ask about teeth, e.g., “is there anything on my teeth?”, ChatGPT can successfully detect. More output examples are shown in Table 3.
Table 3. Some example outputs from ChatGPT given a user-provided photo. The second row includes the specific prompts we used that prompted the model to pay attention to a specific area, and the third row shows the feedback generated by AI.
Table available in the original PDF paper.
[1] [2] [3] Specific prompt: Do you notice anything in this photo of me? Pay close attention to my makeup and its noticeability. Is there anything I need to clean up? Also make sure you tell me these things in regard to MY lefts and rights.
Specific prompt: Do you notice anything in this photo of me? Pay close attention to the area around my mouth or teeth. Is there anything I need to clean up? Also make sure you tell me these things in regard to MY lefts and rights.
Specific prompt: Do you notice anything in this photo of me? Pay close attention to the photo’s lighting and composition. Is there anything I need to clean up? Also make sure you tell me these things in regard to MY lefts and rights. Feedback: It seems like there’s a bit of lipstick on your teeth on your right side, which is quite a common little mishap when applying makeup. A quick check in the mirror should help you clean it up. Everything else looks great! Your smile is lovely and vibrant.
Feedback: Your makeup looks striking and meticulously applied. The eye makeup is particularly dramatic with a bold winged eyeliner that accentuates your eyes. Your eyeshadow is blended smoothly, transitioning from a lighter shade on the inner corners to a darker tone towards the outer edges…
Feedback: In the photo you’ve provided, the backlighting is quite strong, causing the foreground, including yourself, to be in shadow. This is known as being backlit, where the light source is directly facing the camera, making the details of the subject harder to discern. To enhance the photo, you might want to adjust the lighting so that your face and body are more illuminated… [1]https://www.gettyimages.ae/detail/photo/smiling-woman-with -food-particle-stuck-between-her-royalty-free-image/88178261 [2]https://www.businessinsider.com/bridal-wedding-makeup-before-and-after-2019-3 [3]https://rosesandscarsphotography.com/lighting-your-selfie-101-indoor-lighting-overcast-day/
BlindSpot: An AI-Powered Intelligent Mirror Assisting With Facial Hygiene Analysis 277
With a series of prompt engineering, I identified that when the prompt is specific to an area on the person’s face, the model is more likely to make detections correctly. For example, “Do you notice anything in this photo of me? Pay close attention to the area around my mouth or teeth. Is there anything I need to clean up? Also make sure you tell me these things in regard to MY lefts and rights.”, or “Do you notice anything in this photo of me? Pay close attention to my makeup and its noticeability. Is there anything I need to clean up? Also make sure you tell me these things in regard to MY lefts and rights”.
DISCUSSION
Although the AI is able to display a wide range of features and abilities, there are still some limitations. BlindSpot tends to be unable to detect food stuck on teeth and scars. Firstly, it is unable to consistently detect whether there is food on a user’s teeth. This inability needs to be fixed as it is crucial for all the features to be consistently working and able to detect. Another limitation would be the AI’s inability to detect scars. This is important as sometimes, when presented with a picture of someone with scars on their face, the AI would detect it as makeup, or a birthmark, giving false feedback which can limit a user’s experience.
There are also some additional features that can be added to BlindSpot in order to improve its design and user support. One feature could be uploading a video instead of a picture, as in videos, there are quite a few more details that can be missed when using only one picture. This can also improve the ease of use of the users, as they are not required to take “perfectly” framed photos, instead they can just take a video of their whole face. Another feature that can improve the ease of use could be real time feedback, where while taking pictures, the app can notify the users whether they are out of frame or in good composition.
REFERENCES
Fukushima, Y., Uematsu, H., Mitsuhashi, R., Suzuki, H., & Yairi, I. E. (2011, October). Sensing human movement of mobility and visually impaired people. In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility (pp. 279–280). Fussenegger, F., & Spiel, K. (2022, October). Depending on Independence An
Autoethnographic Account of Daily Use of Assistive Technologies. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 1–6). Holly (2019). A message for young people with a vision impairment. Retrieved from: https://lifeofablindgirl.com/2019/03/06/a-message-for-young-people-with- a-vision-impairment/. HousingLin. Personal Stories. Retrieved from: https://www.housinglin.org.uk/Topi
cs/browse/sight-loss-home-the-built-environment/personal-stories/. Huber, S., Alieva, A., & Lutz, A. (2022, October). Vibrotactile Navigation for Visually Impaired People. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 1–4).
278 Lu
Kim Tindall. (2019, April). The Challenges in Going Blind and Learning to Live Again. Retrieved from: https://nfb.org/sites/default/files/images/nfb/publications/ bm/bm14/bm1404/bm140414.htm. Li, F. M., Liu, M. X., Zhang, Y., & Carrington, P. (2022, October). Freedom to
choose: Understanding input modality preferences of people with upper-body motor impairments for activities of daily living. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 1–16). Li, F. M., Spektor, F., Xia, M., Huh, M., Cederberg, P., Gong, Y.,... & Carrington, P. (2022, April). “It Feels Like Taking a Gamble”: Exploring Perceptions, Practices, and Challenges of Using Makeup and Cosmetics for People with Visual Impairments. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1–15). Nilisha Mohapatra. (2015, March). 13 Things You May Not Know About Blind
People. Retrieved from: https://www.lifehack.org/articles/communication/13-thin gs-may-know-blind-people.html. Pradhan, A., & Daniels, G. (2021). Inclusive beauty: How buying and using cosmetics can be made more accessible for the visually impaired (VI) and blind consumer. Cosmetics and Toiletries, 136(4), DM4-DM15. Quora. (2018). How much do blind people care about their physical appearance or
fashion? Retrieved from: https://www.quora.com/How-much-do-blind-people-c are-about-their-physical-appearance-or-fashion. Reddit. (2019). Do blind people care about looking attractive? Retrieved from: https://www.reddit.com/r/TooAfraidToAsk/comments/9pd5te/do_blind_p eople_care_about_looking_attractive/. Yairi, I. E., Azuma, Y., & Takano, M. (2009, October). The one octave scale interface for graphical representation for visually impaired people. In Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility (pp. 255–256). Yairi, I. E., Naoe, K., Iwasawa, Y., & Fukushima, Y. (2011, October). Do multi-touch
screens help visually impaired people to recognize graphics. In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility (pp. 237–238). YouTube. Molly Burke. Retrieved from: https://www.youtube.com/@MollyBurkeO fficial.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Interaction and Emerging Technologies (IHIET 2024)
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-33-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional