Keiko Yamamoto, Taichi Fujihara, Ichi Kanaya (2023). Internet of Toys for Measuring Development of Drawing Skills in Support of Childcare Workers. In: Human-Centered Design and User Experience. AHFE Open Access, vol. 114, pp. 524–530. https://doi.org/10.54941/ahfe1004271
During childhood, play is important for promoting the mental and physical development of children. For this reason, those involved with childcare (guardians and childcare workers) need to create an environment suitable for child development and provide children with support and guidance. However, because there are unique elements to the development of each individual child, childcare workers caring for large numbers of children, and guardians in remote areas, may struggle to oversee the daily development of the children in their care. Especially, in Japan, the lack of vacancy at nursery schools and the number of nursery teachers is regarded as a problem in spite of declining birth rates. Therefore, there is a strong desire to realize a society in which children can be raised safely and securely without placing a burden on parents and childcare workers. To solve the problem, we propose various types of toys with built-in sensors are used to acquire motion data during play activities, and a system intended to aid in estimating the child development stage by creating a data visualization for childcare workers. In this paper, we built a pen holder type device for measuring development of drawing skills. Using this prototype, as the first step of an evaluation, we conducted an experiment to verify whether it is possible to discern four types of drawing focusing on changes in drawing skills related to child development. The results showed that they could be identified with 63% (SVM) and 64% (RF) accuracy.
Japan’s rapid decrease of birth rate leads Japanese to strong wish to realize a society where children can be raised without stresses of parents or child- care workers. While the declining birth rate is on the rise, the lack of vacancy at nursery schools and the number of nursery school teachers is regarded as a problem, and the Ministry of Health, Labor and Welfare has also issued subsidies for ICT conversion of nurseries. However, in the actual sites of childcare, there is still a great deal of anxiety about the introduction of ICT due to changes in work content, increase in workload, and lack of IT literacy. Due to this background, many monitoring systems have been proposed, such as detecting and notifying incidents through collecting big data by attach- ing sensor devices to children. By introducing such systems, it is possible to
Internet of Toys for Measuring Development of Drawing Skills 525
reduce the burden on nursery school teachers by automating daily measure- ment work such as body temperature measurement and to detect and predict accidents automatically. Many of the sensors used for these systems work while in contact with the skin (Suzuki et al., 2017). However, children of an age who begin to have an interest in their own body tend to dislike the sit- uation of sensor’s contact with the body, and when they begin to play, such sensors can be destroyed or accidentally swallowed. For this reason, many of these systems limit their target to management during sleeping and for babies with small physical movement. In early childhood, playing is important to promote mental and physical development. Therefore, the childcare workers must prepare the environment according to the child’s developmental stage and provide suitable assistance and guidance. However, it is difficult to deter- mine the child’s developmental stage, especially for a nursery who watches over many children.
To support them, we propose IoTs (Internet of Toys) with built-in sen- sors that estimates the developmental stage based on the measurement of the movement in the child’s playing in order to support the understanding of the child’s developmental stage (see Figure 1). By embedding sensors and wire- less communication devices into toys that the child play with, they measure how and which toys the child are playing with based on these collecting data and machine learning (deep learning), and estimate children’s physical ability levels. Similar to the present study, prior studies have evaluated the devel- opment of children through play using toys with built-in sensors. Kamide et al. (2018) used building blocks with built-in acceleration sensors, and play with the modified building blocks was quantified for children aged 2 to 6; it was shown that the children’s ages could be estimated based on the data as an index reflective of their cognitive development. In addition, Miron- cika et al. (2018) developed a block with a built-in acceleration sensor that can be used to detect developmental delays in motor skills early without the need for professional help. However, both studies evaluated only one type of play or ability. In the present study, by incorporating various sensors into an assortment of items such as balls, stuffed animals, and writing tools, we aim to estimate the developmental stage of a child from multiple perspectives, rather than using data obtained from only a single form of play. Furthermore, in recent years, studies using machine learning to evaluate motion have been conducted. In Wang et al. (2018), the researchers looked at the learning of effective spiking in volleyball, and estimated the skill levels of different spikes using machine learning. In the present study, following in the footsteps of Wang et al.’s approach, we used machine learning to automatically estimate the child’s stage of development.
So far, we developed a ball-type device using a built-in acceleration sensor as a prototype of a toy with a built-in sensor (Yamamoto et al., 2020). As the result of an experiment, it could distinguish five types of ball activities in ball-throwing movements related to child development with approximately 70% accuracy based on two types of learning algorithms (SVM and RF). In this paper, we propose a pen-holder type device for measuring development of drawing skills.
PROPOSED SYSTEM To help childcare workers assess the stages of childhood development, we propose a system that supports the estimation of the developmental stages in children by presenting data on what toys they play with and how they play with them as measured in an unrestricted and real-time manner using toys incorporating sensors and wireless communication devices. Figure 1 shows an overview of the proposed system, which is divided into the following three parts:
Figure 1: Overview of proposed system.Figure 2: Prototype of pen-holder type device.
Measurement: A toy with a built-in sensor is used to conduct the mea- surements. In this part of the system, movement data from the children’s play (acceleration, angular velocity, pressure, etc.) are continually being sensed, and when a specific condition is detected, that particular portion of the sensor data and the ID of the child are sent to the processing unit through a wireless transmission.
Deep Learning: In this processing part of the system, a learning model is constructed for each toy using various movement data acquired during
Internet of Toys for Measuring Development of Drawing Skills 527
play, and the play and usage of the sensor data received from the measure- ments are estimated. The child’s ID, the date, and the time are all recorded in the database.
Visualization: This presentation part of the system takes the recommended play and usage data related to development collected during the deep learning stage, and creates a visualization of the time sequence for each child for application by the user (e.g., a childcare worker).
For measurement part, we developed new pen-type device attached to crayon, because one of major drawing tools for children in Japan is a crayon. It has a triple-axis MMA8452Q accelerometer was installed along with an Arduino MKR1000 and its battery as a wireless communication device capa- ble of transmitting over Wi-Fi, and a weight as a balancer. The appearance and internal configuration of this device are shown in Figure 2.
EXPERIMENT Rosenbloom and Horton (1971) systematically examined drawing manipu- lations while young children were free drawing and showed that drawing manipulations changed with increasing age. To test the feasibility of the pro- posed system, we used the pen-holder type device described above to see whether specific drawing actions related to these developmental stages of drawing manipulations could be recognized. Based on Kellogg’s analysis of developmental relationship between drawing and manipulation of children (Kellogg, 1969), we came up with the following specific drawing stages (see Figure 3). These four drawing activities are designated as tasks 1–4, respectively.
1. Scribble 2. Diagrams and Combines 3. Aggregate 4. Painting-like
Figure 3: Developmental stages of drawing and task examples.
We confirmed that the participants (five male, ages 21 to 23) were able to perform the experimental task without any problems, and presented them with four sample pictures of the tasks and four videos as models of infants’
drawing actions on each stage. The participants performed like the model drawings, and their drawing motions were collected and video-recorded. Each task was repeated 10 times.
Figure 4: Examples of acceleration data.Figure 5: Confusion matrixes for SVM and RF.
RESULT AND DISCUSSION
Using the procedure described above, 200 data points were collected (5 peo- ple × 4 tasks × 10 repetitions). Figure 4 shows examples of the acceleration data collected for each task. For these 200 data, the identification rate from the results of conducting the leave-one-out cross-validation was 63% for SVM and 64% for RF. In addition, Figure 5 shows the confusion matrix used by the learning model applying SVM and RF to verify which drawing activity was being conducted. The labels in Figure 4 match the tasks listed in Figure 3, where the vertical axis shows the accurately guessed labels, and the horizon- tal axis indicates the predicted labels. This confusion matrix is for the case of LOOCV (leave-one-out cross-validation). In the confusion matrix, task 1 was less misidentified, and for task 2, a minor misidentification occurred. Tasks 3 and 4 showed approximately the same degree of misidentification.
Internet of Toys for Measuring Development of Drawing Skills 529
Through this experiment, using two types of learning algorithms that are considered to be highly accurate, we determined the verifiability of the drawing motions, and in each case, the accuracy of the discrimination was approximately 60%. From the confusion matrix shown in Figure 5, it can be seen that there are less misidentifications in tasks 1. This is because task 1 tends to be most noizy-like and characteristic data. There are also less misidentifications in Task 2 relatively, because a certain pattern can be found in the acceleration data. On the other hand, it can be seen that there are numerous misidentifications in tasks 4 and 5. This means that there was not much difference in the acceleration data generated by these tasks. In addition to acceleration data, other parameters such as drawing duration time may need to be added to distinguish these patterns.
CONCLUSION
Because play is important for mental and physical development in early childhood, it is necessary to provide suitable help and guidance for play throughout the childhood development stages. However, children develop differently, and it is difficult for childcare workers to track the developmen- tal stages of children on a daily basis. Therefore, in this study, we proposed a support system for estimating the progress of these developmental stages that acquires and visualizes motion data from children’s play using toys with built-in sensors. To make this proposed system a reality, a pen-holder type device with a built-in acceleration sensor was created as a prototype, and using it, we conducted an experiment focusing on drawing motions associ- ated with child development, selecting four such motions to verify whether they could be accurately identified. Based on the results of two types of learn- ing algorithms (SVM and RF) used to identify motions, it was found that both algorithms identified the motions with an accuracy of approximately 60%.
To improve the accuracy, it will be necessary to redesign the pen-holder type device such that the sensor is positioned at its holding position. In addi- tion, sensors for determining the angular velocity and other factors allowing the accuracy to be reverified should potentially be added. Furthermore, future issues that need to be addressed include implementing other prototypes such as stuffed animals, designing a GUI for visualization, and determining whether such devices are useful in supporting childcare workers in estimating the stages of development of the children under their care.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Number JP18K11396.
REFERENCES
Kamide H., Takashima K., Ishikawa M., Adachi T. and Kitamura Y., (2018), “Quantitative Evaluation of Children’s Developmental Stage in Building Block Play”, The Transactions of Human Interface Society, Vol. 20, No. 1, pp. 107–114 (in Japanese). Kellog, R. (1969) Analizing children’s art. California: National Press Books.
Mironcika, S., de Schipper, A., Brons, A., Toussaint, H., Kröse, B., and Schouten, B. (2018), “Smart toys design opportunities for measuring children’s fine motor skills development”, In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, pp. 349–356. Rosenbloom, L., and Horton, M. E. (1971), “The maturation of fine prehension in
young children”, Developmental Medicine & Child Neurology, Vol. 13, No. 1, pp. 3–8. Suzuki S., Kurashima S., Hikosaka S. and Abe N., (2017), “Watching Over System for Children in a Kindergarten based on a Human-monitoring Technology”, IEEJ Transactions on Electronics, Information and Systems, Vol. 137, No. 1, pp. 46–53 (in Japanese). Wang, Y., Zhao, Y., Chan, R. H., and Li, W. J. (2018), “Volleyball skill assess-
ment using a single wearable micro inertial measurement unit at wrist”, IEEE Access, Vol. 6, pp. 13758–13765. Yamamoto K., Matsumoto K., Usui T., Kanaya I. and Tsujino Y. (2020), "Internet of Toys for Measuring Ball Play Skills Development to Support Childcare Workers", 22nd International Conference on Human-Computer Interaction (HCII) 2020, Distributed, Ambient and Pervasive Interactions. Lecture Notes in Computer Science, Vol. 12203, pp. 689–698.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human-Centered Design and User Experience
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-90-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional