Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

The Automatic Recommendation and Color Testing Method for Lip Gloss Based on Image Recognition

1 Southeast University, Nanjing, Jiangsu Province, 211189, China ↗
2 Southeast University Nanjing, Jiangsu Province, 211189, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1001092

Abstract

Make-up already became modern day by day, especially a thing of indispensable in female daily life, appropriate make-up look can foil facial features, promote temperament, enhance self-confidence. Lip gloss is the very important part in makeup look. With the continuous development of industrial integration, personalized recommendation of lip gloss for users can improve the consumption experience of users. Based on the investigation and analysis of various factors of makeup, the paper designed a lip gloss automatic recommendation and color testing method based on color temperature recognition of skin color image. In the algorithm design, face skin color recognition is carried out based on convolutional neural network, and then the classifier is used to extract and segment the regions of interest to the face——the lips, and then the lip is colored according to the skin color judgment result. In the paper, experiments were designed to explore users' cognition of the color of lip gloss corresponding to skin color, and experiments were carried out to compare the recognition results of subjects and computers to verify the feasibility of the algorithm designed in this paper.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

INTRODUCTION

Makeup can change the contrast intensity of faces and deepen the three-dimensional sense of facial features. In the overall makeup, lip gloss is very important. Lip gloss can affect the perception of human faces in many ways. On the one hand, different lip colors can make people have the illusion that facial skin color tends to lip color (Hideaki, 2016). On the other hand, the color of lip gloss deepens the contrast of faces, making a more vibrant and energetic perception of the face (Aurélie et al. 2017, Jones, Porcheron, Sweda, Morizot and Russell, 2016).

Combining the above information, it is very important to choose the right lip gloss color. Human skin color can be roughly divided into two categories: cool white skin and warm yellow skin. For different skin colors, suitable lip gloss colors are different accordingly. People with cool white skin are suitable for a wide range of lipstick colors, such as pink and red. People with warm yellow skin are more suitable for warm lipstick colors, such as orange.

This research designs an algorithm to realize the functions of face recognition and lip gloss color testing. The algorithm intelligently recommends lip gloss colors to users through machine learning, so as to improve the user experience of lip gloss consumers. In order to ensure the feasibility of the algorithm, experiments are designed for verification.

DESIGN OF ALGORITHM PROCESS

Figure 1. Flow chart of lip gloss automatic recommendation and color test method based on image recognition The algorithm process of lip gloss automatic recommendation and color test based on image recognition is shown in Figure 1. First, the system will process the input face
Original PDF Figure 1.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 image in two ways. One is to use the image classifier to identify the mouth area in the input image, and remove the skin color pixels in the mouth area and retain the lip pixels in combination with the HSV skin color model. The second is to recognize the input image and determine the color temperature type of the skin color in the input image with the help of the convolutional neural network model. Then, the system will automatically push the relevant lip color model according to the color temperature type of the face in the input image, and search the relevant lip color information from the lip color model database. Finally, the RGB value of lip color in the searched lip color information is assigned to the lip pixels in the input image, and the face picture after processing is displayed.

FACE COLOR TEMPERATURE RECOGNITION METHOD The current image recognition methods can be divided into two categories, one is the traditional image recognition method (Lowe, 2003, Bay, Tuytelaars and Van, 2006, Rublee, Rabaud, Konolige and Bradski, 2011), and the other is the image recognition method based on convolutional neural network (Lecun and Bottou, 1998, Krizhevsky, Sutskever and Hinton, 2012, Simonyan and Zisserman, 2014, He, Zhang, Ren and Sun, 2016). The traditional image recognition method is mainly divided into two steps. The first is to extract the features of the image data, and then a classification model needs to be trained based on the extracted features. The corresponding features are extracted in the online recognition stage and input into the classification model to complete the image recognition. However, such methods are difficult to extract deep feature information, which will have a certain impact on the recognition accuracy. Compared with traditional image recognition methods, the image recognition method based on convolutional neural network can extract deeper abstract features, and can ensure that the system recognizes images more accurately. Therefore, the image recognition method based on convolutional neural network is selected in the paper to recognize the color temperature of the face skin color of the input image. The network can recognize the input image and output the skin color temperature of the face as cool white or warm yellow. The architecture of the convolutional neural network is shown below (see Figure 2).

Figure 2. Convolutional neural network architecture diagram
Original PDF Figure 2.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

The specific implementation steps are as follows:

1) The input layer of the convolutional neural network in this paper stores the color image of the face, and the data depth is 3. There are two types of face image data: cool white and warm yellow (see Figure 3).

2) The convolutional layer will perform convolution operations on the input image data to achieve the feature combination of the input data.

3) The pooling layer reduces the data volume of each layer by compressing the data of each input layer, effectively reducing the data volume input to the lower layer.

4) Synthesize each input feature in the fully connected layer. 5) The features output by the full connected layer are input to softmax classifier for classification, and the probability that the image belongs to each category is output, then the category with the highest probability is taken as the category of the image.

Figure 3. Input image data
Original PDF Figure 3.

IMAGE SEGMENTATION AND COLORING METHOD FOR LIPS In order to overlay the recommended lip color into the lip image region of the face image, the lip image needs to be segmented from the original image. In this paper, the Haar-like classifier (Viola and Jones, 2001) is used for lip recognition. The classifier uses Haar-like features to calculate the difference of pixel values in the region through the feature rectangle, and uses the fast algorithm of integral image to quickly process the required feature values. Then through iterative training through the AdaBoost algorithm, a strong classifier that distinguishes between lips and non- lips is obtained. Finally, the cascade is used to combine multiple strong classifiers to process image information. In order to recognize lips more accurately, a large number of samples are needed to train the classifier. The face image (Figure 4(a)) and the recognition result by trained Haar classifier (Figure 4(b)) are shown below.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Figure 4. Image sample
Original PDF Figure 4.

After obtaining the lip image area, this part can be segmented separately, and the skin color pixels can be confirmed with the HSV skin color model to avoid covering the lip color beyond the lip area. HSV describes the color model through the combination of brightness, saturation and hue. Since the hue value of skin color is relatively concentrated in the HSV space, only considering hue parameters can reduce the impact of illumination on the recognition result of skin color pixels, so that skin color and background can be well distinguished under different lighting conditions. The effect of lip image segmentation is shown in Figure 5(a), and the effect of removing skin color pixels and retaining lip pixels is shown in Figure 5(b).

Figure 5. Image processing effects
Original PDF Figure 5.

EXPERIMENT The purpose of this experiment is to explore the applicable population of lip gloss— —female users' cognition of skin color and applicable lip gloss color, to verify the feasibility of the designed lip gloss automatic recommendation and color test method based on image recognition.

Experimental Preparation Twenty women aged 20-30 were selected to participate in the experiment.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Participants were required to have more than one year of makeup experience and a certain cognitive understanding of lip gloss. All participants have normal color vision and no color blindness or color weakness.

400 female faces were selected as samples from the CelebA dataset, and the 400 samples were randomly divided into 20 groups with 20 images in each group.

Experimental Process Before the experiment begins, the participants are informed of the experimental purpose and the experimental items required to be completed by the participants.

The experiment was carried out in a soundproofed and well-lit laboratory. Participants were required to sit on a designated table and watch sample images on a 17-inch CRT monitor with a resolution of 1024*768 pixels.

The 20 sets of sample images were randomly assigned to 20 participants, and the participants were required to complete 20 sets of experiments. In each set of experiments, the participants needed to judge the skin color as cool or warm based on the face images presented on the monitor. The experimental interface is shown in Figure 6. Each image stays on the display for 10s. If the participant makes a judgment within 10s, the next image will be presented. If the subject does not make a judgment within 10s, the image will be skipped and the next image will be entered.

Figure 6. Experimental interface Experimental Result Analysis
Original PDF Figure 6.

The skin color judgment result of each participant in the experiment was compared with the recognition result of the computer's corresponding image sample, and statistical analysis was performed. The analysis result is shown in Figure 7. Cool color is represented by 0, warm color is represented by 1, the solid blue line represents the judgment results of the participants, the dotted orange line represents the computer recognition results. It can be seen that the computer algorithm has an error, but it is

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7 within the allowable range. The image (Figure 7) is the statistical results of one of the 20 subjects, and the experimental results of all 20 subjects are statistically analyzed.

Figure 7. Comparison of skin color judgment results of image samples Based on the judgement results of the subjects, the accuracy of the computer recog- nition results can be calculated. There are 20 subjects in total, so a total of 20 sets of accuracy data are obtained (see Figure 8(a)).
Original PDF Figure 7.
Figure 8. Statistics of computer recognition accuracy based on experimental subjects The 20 sets of accuracy data are statistically analyzed, and the accuracy above 90% is valid data, while the accuracy below 90% is invalid data. The statistical data are shown above (see Figure 8(b)). The average accuracy rate of all accuracy data is 90.5%, indicating that the algorithm designed in this paper is feasible.
Original PDF Figure 8.

Display of Algorithm Automatic Recommendation Effect The following figures show the application effect of the algorithm designed in this paper. The input face image (Figure 9(a)) and the lip gloss recommendation output by the algorithm (Figure 9(b)) are shown below.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Figure 9. Effect display
Original PDF Figure 9.

CONCLUSIONS

Makeup can enhance the temperament of a person, and lip gloss is a very significant item in the makeup that enhances the temperament effect. For this reason, the paper designs an automatic recommendation and color test method for lip gloss based on image recognition. According to the user's face image, skin color recognition is performed, and according to the skin color recognition result, the corresponding lip color test scheme is recommended. In order to verify the feasibility of the algorithm, this paper designed an experiment to test the accuracy of the algorithm recognition. The recognition accuracy is above 90%, which proves the feasibility of the algorithm.

However, this algorithm still has shortcomings, which are as follows:

1. The number of samples for convolutional neural network training is small. The network can recognize sample data well and has a certain generalization ability, but there will be recognition errors, and the accuracy rate needs to be further improved.

2. The automatic selection of lip pixel points is not accurate enough, resulting in the situation that the lip gloss color is applied to the non-lip pixels, and the display effect of the color test is poor.

Although the algorithm in this paper can recognize the skin color temperature and automatically recommend lip gloss colors, while lip color is not only distinguished by cool and warm colors, but also by texture, such as matte, pearly, satin and highlight. Lip glosses of different textures will also show different effects for makeup. The algorithm designed in this paper can be further improved by adding texture elements on the basis of color.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

ACKNOWLEDGMENTS

This work was supported jointly by National Natural Science Foundation of China [No. 72171044, 71801037], Aerospace Science Foundation of China [No. 20200058069002], Southeast University "Zhi Shan Qing Nian Xue Zhe" Support program funding.

REFERENCES

Aurélie, P., Emmanuelle, M., Frédérique, S., Liu, Y., Ge, L., and Olivier, P., et al.

(2017). Facial contrast is a cross-cultural cue for perceiving age. Frontiers in Psychology, 8, 1208. Bay, H., Tuytelaars, T., and Van. Gool, L. (2006). SURF: Speeded Up Robust Features. Computer Vision and Image Understanding, Graz, Austria, 404-417. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image

Recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 770-778. Hideaki, T. (2016). Facial cosmetics exert a greater influence on processing of the mouth relative to the eyes: evidence from the n170 event-related potential component. Frontiers in Psychology, 7, 1359. Jones, A.L., Porcheron, A., Sweda, J.R., Morizot, F., and Russell, R. (2016).

Coloration in different areas of facial skin is a cue to health: the role of cheek redness and periorbital luminance in health perception. Body Image, 17, 57-66. Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in neural information processing systems, 25(2), 84-90. Lecun, Y., and Bottou, L. (1998). Gradient-based learning applied to document

recognition. Proceedings of the IEEE, 86(11), 2278-2324. Lowe, D. (2003). Distinctive image features from scale-invariant key points.

International Journal of Computer Vision, 20, 91-110. Rublee, E., Rabaud, V., Konolige, K., and Bradski, G.R. (2011). ORB: an efficient alternative to SIFT or SURF. IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011. IEEE. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for

large-scale image recognition. Computer Science. Viola, P.A., and Jones, M. J. (2001). Rapid Object Detection using a Boosted Cascade of Simple Features. Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. IEEE.

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems https://openaccess.cms-conferences.org/#/publications/book/978-1-7923-8988-7

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2022): Integrating People and Intelligent Systems

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-7923-8988-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →