Logan Hansen, Maya Conway, Cody Byrd, Quinn Au, Tauheed Khan Mohd (2024). Comparison Analysis and Evaluating the Accuracy of Pixels Observed using Tobii Eye Tracker 5. In: Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems. AHFE Open Access, vol. 119, pp. 736–747. https://doi.org/10.54941/ahfe1004538
The Tobii Eye Tracker 5 is industry-leading technology in the eye tracking space. Although originally created for gaming, technology can have a big impact in many other fields. For example, it could also be used for people with disabilities to give them access to the web. It also has other possibilities in the medical field, education, help with web development, and many more fields. We also researched and tested the accuracy and precision of the Tobii eye tracker 5. We looked into how different data points form different users doing the same test. For the experiment, we calibrated the Tobii eye tracker for each individual person. From there we took data from two experiments. The first experiment measures the average distance in pixels from where the eye tracker thought the target was versus where it really is. For the second experiment, we collected data from one target and saw how precise to tracer is rather than accurate. We also have referenced a study that was looking at the same types of data however their experiment had a wide range of ages in the experiments.
Most of the knowledge that humans obtained from the external world is gath- ered by our visual senses. Eye contact and gaze directions are very important elements in human communication, indicating our visual interest. Moreover, gaze behavior represents cognitive processes and can express our thinking and intention to a certain extent (Majaranta and Bulling, 2014).
Eye tracking refers to the process of tracking the position of the eyes while they travel across visual scenes (texts, graphics, videos, etc). It uses sensor technology to accomplish this task, which enables the eye-tracking device to identify the distribution of visual attention regarding what we pay attention to, for how long, and in what order. Therefore, eye tracking can provide data concerning perceptual and cognitive processes underlying a variety of tasks at a much broader scale and detail than a verbal representation could do. Eye tracking is being used in various fields from psychology and medical
Comparison Analysis and Evaluating the Accuracy of Pixels Observed 737
diagnostics to interactive studies, and entertainment usage (Scheiter and van Gog, 2009).
Initially, eye movements were mainly studied by physiological introspec- tion and observation. Basic eye movements were sorted and the duration of the eye activities enabled precise measurements of eye movements. The first-ever eye-tracking system was highly uncomfortable and not practical. A break-through in eye-tracking technology was the development of the first eye-tracking system in the early 1900s using photography and light reflected from the cornea. The development of an unobtrusive eye-tracking camera- based system and the rise of computing power enabled the ability to gather eye-tracking data in real time (Majaranta and Bulling, 2014).
Figure 1: Demonstration of precision versus accuracy.
Today, there are many different techniques to track and record eye move- ments. The most popular technique used by modern eye-tracking systems is Pupil Center Corneal Reflection (PCCR) (Punde et al., 2017). It uses a near- infrared camera or other optical sensors for tracking the gaze direction. To better understand, near-infrared light is directed towards the pupil - the cen- ter of the eye that causes visible reflections in the cornea, which are tracked by a camera. The eye image that is captured by the camera will be used to identify the reflections of the light source on the cornea and pupil. Then, the system will calculate the vector formed by the angle between the cornea and pupil reflections, and from this, gaze direction will be determined. The vector calculation method includes the following:
Velocity based
Dispersion based
Area of interest A study was done with the Tobii TX300 eye tracker aimed to discover the different levels of accuracy and precision of the technology with a wide range of participants. Participating in the int experiment were 36 18-month-old toddlers (24 female), 11 children (7 female), and 11 adults (10 female, mean age 26.6 years). They ran an experiment displaying five points on a screen and tried recording each user’s look at all the points. In the experiment, only the children and adults were given instructions while the toddlers were not.
For the toddlers, they used 28 tests due to the eye tracker missing information and outlier data. They said, “For individual targets, the accuracy was 0.16
at best, and 12.16
at worst. For our precision measures, the average SD across participants was 0.20
, 0.21
. The mean RMS was 0.25
, 0.25
” (Clemotte et al., 2014). When they looked at all of the data from all of the groups they discovered a lot of the data points from the models had a higher error percentage. However, there is no difference between children to adults using the eye tracker.
Figure 2: Figure of the Tobii 5 eye tracker hardware that was used in the experiment.
RELATED WORK
Eye tracking technology has come very far since its inception in the early 1900s for us to be here in our class with the ability to use a device as sophis- ticated and intricate as the Tobii Eye Tracker 5 solely for a class project. Prior to devices like the one we are using today, the user would have to wear some type of headgear, or even have to insert something into their eye in order to properly track their gaze. That made the technology extremely expensive and restrictive in their movement, along with it being overall inaccessible to the majority of the general public. This then becomes a large hindrance to being able to conduct proper studies and acquire accurate research, also limiting the device’s accuracy and reliability when it comes to conducting research.
Based on the research we have done, there have been numerous studies that highlight the application of eye tracking in various fields. These include things like gaming, software engineering, marketing, art, psychology, medicine, web design, and biometrics.
Computer Gaming Erika Jönsson researched how eye-tracking technology has been implemented in the area of computer games. First, she explores the eye-tracking technology itself, looking into how this eye-tracking technology has begun to be used in computer games. She proceeds to give an in-depth analysis of what the tech- nology is actually capable of when it comes to computer games. She referred to a focus group where participants were given Tobii eye-tracking technol- ogy and shown a demonstration of the technology in active use. When asked about their thoughts on their experience with the technology, the partici- pants gave some personal requirements that an eye tracker should be; fast,
Comparison Analysis and Evaluating the Accuracy of Pixels Observed 739
and accurate, the device should not be noticeable, and it should be simple and easy to calibrate. Through a series of tests, focus groups, and related medi- ums, companies have refined the eye tracker technology to things like the Tobii Eye Tracker 5 which our group is using today. By using this new and improved eye-tracking technology, gaming has now reached a point where devices like the Tobii eye tracker can be implemented to allow for differ- ent ways the user can interact with the computer to manipulate the game in some meaningful way. The examples given were of first-person shooters where instead of aiming with the mouse the user would instead use the eye tracker to aim where they are looking on the screen.
Along with the uses of the eye tracker that were mentioned by Erika Jönsson in her article, the eye tracker has many other uses in gaming. On top of the user having the ability to have a much more natural targeting sys- tem in first-person shooting games, eye tracking has expanded to all forms of gaming. Ranging from rudimentary rhythm games like Beatshot where you have to look at certain targets on the beat of a song to gain points, to more complicated and complex games where you have to solve puzzles by using the eye tracking system in some unique way. These many different applications of the system help grow the eye tracking system’s popularity and in part its advancement (Jönsson, 2005).
Software Engineering In the field of software engineering, the community has been using eye track- ers since the mid-1990s and they have become increasingly recognized to be useful tools to acquire empirical data. In the article, A systematic litera- ture review on the usage of eye-tracking in software engineering, the authors examine the ways in which researchers have used and are using eye tech- nology in their work. The authors found that researchers used eye tracking for a variety of purposes such as studying model comprehension, code com- prehension, debugging code, collaborative interaction, and traceability. On top of that the authors found that some studies have used various metrics based on eye-movement data to obtain quantitative measures. This all being said, current eye-tracking technology does still have its limitations, and the authors acknowledge that these limitations may threaten the validity of previ- ous studies. That being said, the authors concluded that with the onset of new eye tracking technology, the use of the tools become much easier and overall more user friendly. Making it more to the software engineering communities which overall benefits the eye-tracking technology as a whole.
In the article, the authors also go through different studies that examined how people of varying levels of skill and knowledge in software engineering view code, along with the difference between reading code and normal text. The study they used found that experts pay more attention to beacons in code than novices, while the novices do not discriminate between different areas of the source code when reading. The study also found that the participants spend more fixation times and have higher regression rates when reading the source code compared to natural text. It also showed that the higher com- plexity of the source code, the more it forced the participants to change their
focus of attention more frequently. Specifically, when reading the code partic- ipants paid more attention to identifiers, operators, keywords, and literals, in that order, while separators receive almost no attention. This trend of atten- tion and how people view code is also seen in not only the process of just reading code but also things like the debugging process (Sharafi et al., 2015).
Marketing According to an article by H. Zamani, it has been found that consumers are most likely to purchase a product the more they fixate on it through eye tracking data. In other words, the more someone’s eyes are drawn to an image or product the more likely they are to purchase it. For this reason, eye tracking is helpful in the fields of advertising and marketing. Eye tracking allows researchers to find what colors and designs draw consumers to the product (Zamani, 2016).
Art Eye tracking has been used to track what people pay attention to in muse- ums to provide information for the tourism industry. Eye tracking devices “can provide us with new insights into very individual experiences, appro- priation strategies, and goals of visitors”(Kira Eghbal-Azar). It allows for the collection of data about what people find enticing at museums and allows one to see the museum from different tourist perspectives (Eghbal-Azar and Widlok, 2013).
Psychology There are also applications for eye-tracking devices in psychology. Eye- tracking devices have been used in order to detect schizophrenia and autism. Using eye tracker devices, it has been found that certain atypical gaze pat- terns may be indicative of autism disorders. In particular, when looking at faces gaze patterns differ drastically between individuals with autism and neurotypical people (Dr. Deborah M Riby) (Riby and Hancock, 2008).
Medicine Eye-tracking is being used widely in different medical fields. One of the major research topics is the modeling of visual research patterns. According to State of the Art: Eye-Tracking Studies in Medical Imaging the first eye tracker was made by Edmund Huey in 1908. The first non Intrusive eye tracker was made in 1937 and was used to study cognitive psychology.
One of the other Major fields is search patterns. In the medical field when someone gets an x-ray in cereal areas there is a chance to be 30 percent false positives. So the researchers were trying to look if there was a correlation between the research pattern and the rate of false positives. According to the paper, “Nodine et al. carried out an eye-tracking experiment where the par- ticipants (i.e., three mammographers and six radiology trainees) were asked to view 40 mammogram cases and decide whether they were “normal” or “abnormal”. Their eye movements were recorded using an ASL4000 eye- head tracker. Experimental results showed there was no significant difference
Comparison Analysis and Evaluating the Accuracy of Pixels Observed 741
in terms of the decision time between experts and trainees, however, the per- formance of mammographers was always higher than trainees” (Clemotte et al., 2014). Although the study did not find any real correlation there are many other fields in that research can and will be conducted that could have a correlation (Lévêque et al., 2018).
Driving The automotive industry is looking into the new technology of eye tracking. They are looking at the visual angles and events drivers look at while driving. They have been trying to do this with two different ways of eye tracking. The first is a helmet that the driver wears while driving and the other is a bar in front of the driver that tracks the driver’s eyes. During the experiments, they divided the driver’s field of view into three zones and recorded how many times the drivers looked into each area. They had requirements for the eye tracker to be within a certain tolerance as stated in the paper”. Here are the 5 criteria he distinguishes: – Accuracy – the limit of accuracy of the eye is approximately ±0.5
; most devices meet this criterion; only studies on micro- saccades and rapid eye movements require better accuracy”(Kapitaniak et al., 2015). They had to be able to tell if the information that was collected by eye trackers was within specific tolerances. This accuracy that they need is what our experiment is testing. Without an accurate eye tracker, the experiments can not be completed.
Web Design Another application of eye tracking is seen in the study, Generations Y’s, web design, and eye tracking. In this study, researchers Soussan Djamasbia, Marisa Siegelb, and Tom Tullisb conduct a two-part study to find out how Generation Y (ages 18–31) interact with websites and what they are more likely to be drawn to when browsing the internet. In the first part of the study, they conducted an online survey which was used to find out the kinds of pages that Generation Y enjoys. After conducting the survey, they used an eye tracker on the pages that were most and least liked from the first part of the study. The participant’s eye movement is tracked while browsing the pages, which is used to find evidence of what attracts Generation Y’s atten- tion. The research in this study is slightly similar to that of studies that have to do with things like advertising and marketing. The researchers use partici- pants to try and assess what kind of design is the most appealing to a specific group of people. However, the purpose of this study is to specifically find data as to how Generation Y consumes media on the internet and how they are interpreting what they see on the websites they are visiting. The results are meant to show what kinds of things and what certain aspects of a website are most appealing to those of Generation Y because of their demographic size, containing eighty-two million people and spending about 200 billion dollars annually. The results of these studies showed that Generation Y prefers pages that include a main large image, images of celebrities, not a lot of text, and a search feature (Djamasbi et al., 2010).
Biometrics In the paper, Eye movements in biometrics, authors Pawel Kasprowski and Jo’zef Ober present a new technique of bio-metric identification that is based
on a person’s eye movements characteristic. Through this method of iden- tification, the system measures the person’s eye reaction to certain visual stimulation. The first experiments the authors examined were used to show the possibility of being able to identify a specific person using this exact method. The eye-tracking biometric system like many other technologies has its shortcomings but it does carry with it some significant advantages. It can compile the person’s behavior and their physiological aspects, mak- ing it extremely difficult to copy and counterfeit that data, along with that it is extremely simple and easy to perform the tasks necessary to satisfy the biometrics. This can then be combined with things like facial recognition soft- ware, such as the kind used to unlock a phone using your face, to better lock your devices. The eyes are one of, if not the, most important organs in our body so it only makes sense to build technologies that can work with that organ to better our lives in some possible way (Kasprowski and Ober, 2004).
EXPERIMENT Our experiment is designed to test the accuracy of the eye-tracking device. We are defining accuracy as its ability to track where our eyes are looking when we gaze at a specific point. In order to test the accuracy our design is to use ms paint to measure the distance between a spot where we are gazing and where the eye tracking device registers our gaze. The objective is to create a spot on the screen where we will look at and use the eye tracker in order to track our gaze. Then the distance between the spot we are gazing at and where the gaze is registered will be measured.
Technology Specifications The Tobii experience software was downloaded onto a Dell PC running the Windows 10 operating system with a 3 GHz Intel Core i7-9700 processor with 32 GB of ram and an Nvidia Quadro P620 graphics card. The monitor used was a 24-inch, 1920 × 1080 pixel display Dell monitor. The Tobii 5 eye tracker was connected through a USB port to the computer and mounted on a 3D printed stand and placed at the base of the monitor in front of the participant. The experiment was done using the latest version of Microsoft Paint as of Friday, January 20th, and using the feature in the Tobii Experience software that allows the user to move the position of the mouse to the viewer’s current eye position on the screen by pressing a button that is mapped by the user, which was mapped to Left Control.
Procedure Four participants were involved in this study. All four participants were aged from nineteen to twenty-two. Two of the participants were wearing glasses and the other two did not have glasses or contacts. All of the participants were informed of the experiment and of its material and nature beforehand. The participant sat in front of the computer that was discussed in the portion of the technical specification and began by calibrating the eye tracker by using the Tobii Experience software. The participant was given a Microsoft paint canvas with 10 dots evenly placed throughout as seen in Figure 3.
Comparison Analysis and Evaluating the Accuracy of Pixels Observed 743
Figure 3: One of our participants used the template that was created and used by the participants to acquire the data and complete the experiment.Figure 4: The template that was used for collecting data in the experiment.
The participant then used a feature included in the Tobii Experience soft- ware that allows the user to move the cursor on the screen to the current location of where the user is looking at the screen. This command was mapped to the left control button. The participant was then instructed to look directly at the dot to the best of their ability and then press the left con- trol button. Once the cursor was moved the participant would vocalize the exact pixel location of where the cursor was on the screen and the data was recorded. The participant was then instructed to look to the far right of the screen to reset the cursor and then repeat the process for the rest of the dots on the screen.
After this portion of this experiment was completed, the participants were given a second photo to conduct a similar procedure. They were shown a photo of a target and instructed to look at the center of the target to the best of their ability and record the coordinates of their determined POG. The acquisition of the coordinates was the same as the previous part of the experiment where the user would use to tool in the Tobii Experience software to move the cursor to the users determined POG and the exact coordinates would be recorded. The participant did this 5 times and then was switched out with the next person.
Data Processing To calculate the Euclidean distance between the targeted dot and the deter- mined POG, the approximate accuracy of the sensor can be calculated in terms of the visual angle. The test is done 4 times: 2 times with glasses and 2 times without glasses to compare accuracy between 10 dots: 5 dots on the top row and 5 dots on the bottom row. For the approximate precision of the sensor, an experiment of hitting the bullseye is done in a similar process.
The first test was 10 points done in ms paint and the accuracy was calcu- lated using the Euclidean distance to find the visual angle between the sensor’s determined position and the actual point.
The second test was hitting the bullseye and precision was calculated using the Euclidean distance as well The Euclidean distance is calculated using the x and y coordinates between the determined point and the actual point using the following formula:
Equation reproduced from the original PDF.
The visual angle can be calculated using the Euclidean difference between the two points and the distance of the observer to the screen. The relationship is as follows: Visual Angle = 2 ∗arctan((Difference/2)/Distance)
To evaluate the accuracy, the average Euclidean distance across all test points was calculated using Microsoft Excel, along with the average Euclidean dis- tance across the five top dots and the five bottom dots in the ms paint experiment, and the average Euclidean distance on the bullseye in the target experiment.
To evaluate precision, we assessed the scatter plot that we conducted after doing the target experiment. The scatter plot was made based on the average Euclidean distance of the bullseye over five trials per person.
RESULTS
Our first set of results looks at the Euclidean distance results between the targeted dots and the determined POG. The data is recorded and calculated 10 times for this MS Paint test. Then, we calculated the mean distance (60), minimum distance(148), and maximum distance (31) in pixels to evaluate the accuracy of the eye tracker. There is not a significant difference, however, the difference is enough to conclude that the accuracy of the Tobii Eye Tracker 5 could be improved. Our second set of results looks at the Euclidean distance results across all the 5 trials trying to hit the bullseye. To determine whether Tobii eye tracker 5 has good precision, we looked at the Euclidean results and the targeted bullseye’s location. We conclude that the Tobii eye tracker 5 doesn’t perform consistently. Looking at the results, the difference could be from 35 to 110 without too many particular changes from the methods of conducting this experiment, which shows that the precision of the Tobii eye tracker could be improved.
Comparison Analysis and Evaluating the Accuracy of Pixels Observed 745
MS Paint Trial Results Test User 1 User 2 User 3 User 4
If we continued our work in the future we would spend a lot more time trying to get the Tobii eye tracker 5 integrated into Unity. As of now, there is little information on how to do it and troubleshooting help. We would possibly have to get a newer eye tracker to get it to work constantly. But once it is integrated and working constantly we would build a game similar to the Wii shooting range game from Wii Play. The game would shoot where your eyes are looking whenever the user presses the mouse or the space bar. This could potentially be done by using a wearable eye-tracking device, which would be much more accurate and exact than the Tobii eye tracker 5. Allowing us to not only be able to use the technology in unity but also allow the user to be much more precise in their use of the eye tracker (MacInnes et al., 2018).
CONCLUSION
In conclusion, the Tobii eye tracker 5 is a device that still has a lot of poten- tial for its growth that we can all look forward to in the upcoming years. Eye tracking is the process of tracking the position of the eyes while they travel across visual scenes (texts, graphics, videos, etc). Eye tracking has been applied to various different fields, in our paper, we have discussed its applica- tions in computer gaming, software engineering, marketing, art, psychology, medicine, driving, web design, and biometrics.
However, the lack of software and some hardware limitations restricts some of the possibilities with the technology. For our project, we tried to create a game using Tobii’s TobiiSDK, however, we were unable to get the eye tracker to connect properly and work with the Unity Development kit. We also had difficulties with the tracker losing connection in the middle of use. When we calibrated the eye tracker for every person in the exper- iment, it still struggled to be accurate and precise. Despite its limitations right now, the Tobii eye tracker still proves to be a promising device that could be improved given time and investment. A study was done to see what happens when the user moves their head or covers their eyes and recorded what happened, they found some latency in the results when using T60XL (Niehorster, 2018). We think this technology in a few years will be used in a large number of different fields and will continue growing and being developed.
REFERENCES
A. Clemotte, M. A. Velasco, D. Torricelli, R. Raya, and R. Ceres Ruiz, “Accuracy and precision of the tobii x2-30 eye-tracking under non ideal conditions,” 2014. B. Kapitaniak, M. Walczak, M. Kosobudzki, Z. Jó´zwiak, and A. Bortkiewicz, “Appli-
cation of eye-tracking in drivers testing: A review of research,” International journal of occupational medicine and environmental health, vol. 28, no. 6, 2015. D. C. Niehorster, T. H. Cornelissen, K. Holmqvist, I. T. Hooge, and R. S. Hessels, “What to expect from your remote eye-tracker when participants are unre- strained,” Behavior research methods, vol. 50, pp. 213–227, 2018. D. M. Riby and P. J. Hancock, “Viewing it differently: Social scene percep-
tion in williams syndrome and autism,” Neuropsychologia, vol. 46, no. 11, pp. 2855–2860, 2008. E. Jönsson, “If looks could kill–an evaluation of eye tracking in computer games,” Unpublished Master’s Thesis, Royal Institute of Technology (KTH), Stockholm, Sweden, 2005. H. Zamani, A. Abas, and M. Amin, “Eye tracking application on emotion analysis
for marketing strategy,” Journal of Telecommunication, Electronic and Computer Engineering (JTEC), vol. 8, no. 11, pp. 87–91, 2016. J. J. MacInnes, S. Iqbal, J. Pearson, and E. N. Johnson, “Wearable eye-tracking for research: Automated dynamic gaze mapping and accuracy/precision comparisons across devices,” BioRxiv, p. 299925, 2018. K. Eghbal-Azar and T. Widlok, “Potentials and limitations of mobile eye track-
ing in visitor studies: Evidence from field research at two museum exhibitions in germany,” Social science computer review, vol. 31, no. 1, pp. 103–118, 2013. K. Scheiter and T. van Gog, “Using eye tracking in applied research to study and stimulate the processing of information from multi-representational sources. themed issue of applied cognitive psychology, 23 (9),” 2009. L. Lévêque, H. Bosmans, L. Cockmartin, and H. Liu, “State of the art: Eye-tracking
studies in medical imaging,” IEEE Access, vol. 6, pp. 37023–37034, 2018. P. A. Punde, M. E. Jadhav, and R. R. Manza, “A study of eye tracking technology and its applications,” in 2017 1st International Conference on Intelligent Systems and Information Management (ICISIM), pp. 86–90, IEEE, 2017. P. Kasprowski and J. Ober, “Eye movements in biometrics,” 2004. P. Majaranta and A. Bulling, “Eye tracking and eye-based human-computer interac-
tion,” in Advances in physiological computing, pp. 39–65, Springer, 2014.
Comparison Analysis and Evaluating the Accuracy of Pixels Observed 747
S. Djamasbi, M. Siegel, and T. Tullis, “Generation y, web design, and eye tracking,” International journal of human-computer studies, vol. 68, no. 5, pp. 307–323, 2010. Z. Sharafi, Z. Soh, and Y.-G. Guéhéneuc, “A systematic literature review on the usage
of eye-tracking in software engineering,” Information and Software Technology, vol. 67, pp. 79–107, 2015.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-95-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional