Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

A comparison of accessibility assessment tools in the inspection of an informative government website on the situational status of COVID-19

Pontificia Universidad Católica del Perú San Miguel, Lima 32, Peru ↗
Publication year
Digital Object Identifier10.54941/ahfe1001647

Abstract

Accessibility is considered a highly relevant quality aspect of software products. However, few websites are designed considering minimum standards that make applications accessible to all people regardless of their capabilities and abilities. Considering that there is a representative percentage of people who experience some form of disability, it is a requirement that government websites that provide information to their citizens meet an appropriate level of compliance. In this study, we report the results obtained from using three accessibility evaluation tools to examine an informative Peruvian government website on the situational status of COVID-19. In addition, the results were compared to determine the differences between the tools. Despite covering the same WCAG standard, the findings demonstrate that the tools evaluate different aspects. Likewise, despite being a government site and being required to comply with a minimum standard of accessibility, the evaluated website lacks some attributes to be accessed by any citizen.

INTRODUCTION

Many factors today determine whether one software product is better than another. One of these aspects is accessibility, which refers to the capability of an application to be used by anyone regardless of their abilities or skills (Petrie & Kheir, 2007). This quality attribute has become highly important and is currently a concern of both companies and software development teams due to the increasing and the representative number of users who experience some disability. According to the World Health Organization (World Health Orga- nization, 2021), one billion people, or 15% of the world’s population lives with some form of disability. In the United States, 26% (one in 4) of adults have some type of disability (Centers for Disease Control and Prevention, 2020). In Peru, where this research took place, 10.3% of the population (around 3 million people) experiences some disability, according to the census

© 2022. Published by AHFE Open Access. All rights reserved. 94

A Comparison of Accessibility Assessment Tools 95

carried out in 2020 by the National Institute of Statistics and Informatics of Peru (INEI) (Instituto Nacional de Estadística e Informática - Perú, 2021). The high numbers show that there is a significant group of citizens that must be considered when designing software products, especially if government entities provide these technological tools.

Web accessibility is a fundamental right of any person. According to United Nations (United Nations, 2006), the Web must be accessible to provide equal access and equal opportunity to people with diverse abilities. In the same way, there are regulations and laws in different countries that require companies and entities to offer accessible websites and software products. This aspect becomes even more relevant in the context of electronic government, referring to websites that provide citizens services and information of interest.

In Peru, there is currently Law N° 29973 that modifies Article 3 of Law N° 28530 (Law to promote Internet access for people with disabilities and to adapt physical space in public Internet booths), which establishes that “public entities and universities must incorporate access options on their web pages or Internet portals so that people with disabilities can access the information these contain”. In the same way, there is a Ministerial Resolu- tion N° 126-2009-PCM that establishes accessibility guidelines for the design of web pages and applications. However, despite existing regulations and policies that should be respected for website design, in a previous study (Paz & Paz, 2021), it was concluded that companies ignore these guidelines and that the gap between what is required and what is offered is still quite wide. This may be because there is little concern on the part of companies to provide quality websites and the absence of a regulation that establishes penalties for companies that do not comply with the laws.

Throughout the period of the pandemic caused by the Covid-19 virus, the Peruvian government has initiated a set of promising initiatives that have allowed the population to be informed about the country’s current state in terms of health. One of these significant initiatives was constructing a web- site that reports minute by minute on the situational status of the virus in the country (Ministerio de Salud - Perú, 2021). On this website, citizens are informed about the total number of positive cases, the number of discard tests applied, the number of hospitalized, the number of deceased people, and the number of vaccinated people, data that can be listed according to search fil- ters. Given the relevance of the information displayed on this site and the importance that the population is informed, it is essential that this website be accessible and that all people can consult the data without creating barriers that exclude them.

In this study, the results of an accessibility assessment are reported to the Peruvian government website that reports on the situation of Covid-19 in the country. Given that the website provides information of public and health interest, is directed to all citizens, and in accordance with national regula- tions, should be accessible to all. This accessibility assessment was carried out through automated tools recommended by the W3C Web Accessibility Initiative. The results were analyzed, discussed, and compared. The intention of this work is that it can serve as a case study reference for other professio- nals and specialists in the area of Human-Computer Interaction and also to

be taken as a precedent in the improvement of websites that are provided to citizens, and in this way, increase the quality of the applications developed.

RELATED WORK

Accessibility inspections can be conducted in different ways. According to a systematic review (Nuñez et al., 2019) that aimed to identify studies that report cases of accessibility evaluations to software products, it was possible to determine that there are three methods: (1) expert evaluation, (2) automatic tools, and (3) user testing. Of these three methods, the use of automated tools is the most reported by specialists due to its simplicity, ease of use, and little resources that it demands. Likewise, according to the findings, it has been identified that there is no method that is better than another (Paz et al., 2021). The methods complement each other, and even the specialists recommend that it is required to apply all the methods in as many scenarios as possible to contrast the results obtained for an exhaustive evaluation.

Accessibility assessment through automated software tools can be per- formed following a methodology defined by the W3C Web Accessibility Initiative called Website Accessibility Conformance Evaluation Methodology (WCAG-EM). This methodology is an approach to determine the degree to which a website complies with the Web Content Accessibility Guidelines (WCAG). The conformance evaluation involves five main steps: (1) define the scope of the evaluation, (2) explore the website, (3) select a representa- tive sample, (4) evaluate the selected sample, and (5) report the evaluation findings (World Wide Web Consortium, 2014). The purpose of this evalua- tion is to verify if the interfaces have been designed following the principles established in the WCAG international standard and to determine if they have elements that allow them to be accessed through assistive technology, such as software and hardware that people with disabilities use to improve their interaction with the web.

The WCAG standard is the most widely used by Human-Computer Inte- raction professionals to perform an accessibility assessment (Benites Alfaro & Zapata Del Río, 2018). These regulations define principles that allow a design without barriers that make it difficult or impossible for some people to use websites. Some examples involve the inclusion of alternative text for images, the ability to use a website not necessarily by mouse but also from the keyboard or assistive technology, as well as the ability to access audio transcripts aimed at people who cannot hear.

The currently valid versions of the WCAG standard are 2.0 and 2.1. They have from 12 to 13 guidelines that are organized under 4 principles: (1) perceivable, (2) operable, (3) understandable, and (4) robust. For each guideline, there are testable success criteria, which are at three levels: A, AA, and AAA. Automated tools can verify the success criteria in complying with the guidelines of the graphical interfaces of a certain website. For this, the URL of the website is required. However, when the application demands a login or uses mechanisms to display graphic components in real-time, it is required to install a plugin attached to the web browser to perform the inspection.

A Comparison of Accessibility Assessment Tools 97

Regarding the conformance levels, Level A is the lowest and the most basic one. If a website needs to comply with Level AA, it is required that everything established by Level A has been previously considered. In other words, con- formance at higher levels indicates conformance at lower levels. Likewise, the appropriate and recommended level for all websites is AA since Level A is very basic and Level AAA success criteria it is not possible to satisfy all for some content.

ACCESSIBILITY EVALUATION SETTING In this study, the accessibility of a Peruvian government website dedica- ted to providing information on the current situation of COVID-19 in the country was evaluated. This website was developed by the government with the aim of offering the population up-to-date and real-time infor- mation about the cases identified and the progress of the immunization process. Likewise, the project sought to support scientific, clinical, and epidemiological research in the context of COVID-19 in Peru, by making open data from the database that collects the official information of the Ministry of Health available to the community in a format standard for analysis.

The website selected for the analysis provides citizens with information on the number of positive cases and the number of deaths from COVID- 19. In relation to the number of positive cases from COVID-19, the website is a means to report on: (1) increase in new cases nationwide, (2) cases by sex and age, (3) confirmed cases and reported in the last day, (4) cases by rapid test, and (5) cases by molecular test. On the other hand, in rela- tion to the number of deaths from COVID-19, the website provides data on: (1) increase in the number at the national level, (2) accounting by gen- der and age group, and (3) deaths in the last day. The purpose of this website is to keep the population informed and serve as an open-source of data for research centers and universities interested in performing statistical analysis.

The URL of the website is covid19.minsa.gob.pe/sala_situacional.asp, and a screenshot of the interface can be seen in Figure 01.

The automated tools that were used for the accessibility evaluation pro- cess were selected from the list recommended by the W3C WAI (World Wide Web Consortium, 2020). Since the website is secure and dynamic, it was required to select tools that can be added as a plugin to the browser to per- form the inspection. The selected tools were: (1) WAVE, (2) axe Dev-Tools, and (3) Siteimprove Accessibility Checker. All these tools are certified by the W3C WAI and allow an evaluation based on the WCAG 2.0 standard. To carry out the accessibility evaluation, it was necessary to install these tools in Firefox directly as an add-on. The sections of the website that were selected for inspection are: (1) the informative home page (covid19.minsa.gob.pe), as well as the sections that show: (2) summary of the data, (3) data on the number of positive cases, (4) data on the number of hospitalized, (5) data on people in intensive care, (6) data on the number of deceased people.

Figure 1: Screenshot of the graphical interface of the informative government website on the current situation of COVID-19 in the country.
Original PDF Figure 1:
Table 1. Number of accessibility issues identified on the website by the tools.

Table available in the original PDF paper.

Section of the Website WAVE axe DevTools Siteimprove

A AA A AA A AA Main page 4 0 1 0 3 1 Summary 1 0 2 0 3 0 Positivity 1 0 3 0 2 0 Hospitalized 1 0 2 0 2 0 Intensive care unit 1 0 4 0 2 0 Deceased 1 0 2 0 2 0 TOTAL 9 0 14 0 14 1

ANALYSIS AND RESULTS OF THE ACCESSIBILITY EVALUATION The accessibility assessment was performed on December 27, 2021. Table 1 shows the results obtained from the evaluation. Based on the findings, there is a small number of issues that need to be addressed for the website to achieve the level of conformance AA. However, it is currently not possible to men- tion that the website that provides information on the situational status of COVID-19 in Peru is accessible. Likewise, it is not possible to establish that a minimum level of accessibility is met since the identified problems belong to the level of conformance A.

Regarding the results offered by the different tools, some different but minimal ones can be appreciated. Quite coincidences can be appreciated, and the few differences may be due to the way the standard is interpreted by each tool. Depending on the way in which each tool implements the verification of compliance with a guideline, the assessment, as well as the results, could be

A Comparison of Accessibility Assessment Tools 99

Table 2. Top 5 of the identified issues that are most frequent in the interfaces.

Table available in the original PDF paper.

ID Problem description P1 Inline frame without a text alternative: Every inline frame (iframe) should have a text alternative that summarizes its content or purpose (Guideline 4.1.2.) P2 Page language has not been identified: The main language of the page has not been declared within the <html> element. (Guideline 3.1.1.) P3 Image without a text alternative: All non-decorative images must have a text alternative (also known as “alt text”). (Guideline 1.1.1.) P4 Page zoom is restricted: The viewport element on this page prevents users from zooming in or scaling text content. (Guideline 1.4.4. and 1.4.10.) P5 Linked image missing alternative text: If an image is within a link that contains no text and that image does not provide alternative text, a screen reader has no content to present to the user regarding the function of the link. (Guideline 1.1.1 and 2.4.4.)

different. However, the difference is not significant, and it could be concluded that the results are similar.

About the identified problems, the majority are referred to the non- existence of alternative text for the links and images, definition of the website’s language, as well as the unfeasibility of zooming. The five pro- blems with the highest degree of occurrence are detailed in Table 2. From these results, it is possible to determine that although the website does not meet the minimum level of compliance regarding the standard WCAG, the problems are not serious and can be solved without much complication. The findings indicate that at least for this case, they are favorable and promising. In a certain way, considerations have been taken since in previous investi- gations (Paz et al., 2020; Paz & Paz, 2021), the results were not positive, demonstrating a wide gap between what was expected and what was offered. Likewise, it is expected that this type of research can be considered both at a methodological level as a case study and for an improvement in the quality of the software products that are available to people on the Web.

CONCLUSIONS AND FUTURE WORKS Accessibility is an important quality attribute that must be considered now- adays in software product development. According to the censuses, the number of people who experience some type of disability worldwide is repre- sentative and that is why companies and development teams should have considerations when designing software products. The accessibility issue becomes much more relevant when it is related to applications that are developed by government entities to provide citizenship with public servi- ces. These services can range from informational to applications that involve transactions. However, in previous studies, we were able to determine that there is still a gap between the degree of accessibility that websites are expe- cted to have and what they currently provide, specifically in the context of e-commerce. In this study, an accessibility evaluation has been carried

out on a Peruvian government website that provides real-time information about the COVID-19 situation in the country. It is important that this web- site is accessible since it provides relevant information in a way in which the population can make decisions regarding the current state of emergency. Likewise, the mentioned website offers information as an open database since it can be used by universities and research centers in data analysis. The data that can be found in this web application is related to the number of peo- ple who have received the vaccine, as well as those detected with the virus, the number of deceased people, associated with percentages and statistics by region.

The evaluation was carried out using three different authorized automa- ted tools and according to the WCAG 2.0 standard. Based on the results, it was possible to determine that although the minimum level of compliance established by the standard is not met, a small number of aspects must be corrected to reach the AA Level. Also, the automated tools show small dif- ferences between each other because they interpret and perform the review differently. However, the coincidences are many and it could be established that there is no significant difference between the different reports offered. The results obtained with respect to this e-government website differ from previous studies, where the gap between what is expected and what is offe- red is wide. This study can be used as a methodological guide and case study of an accessibility assessment carried out in the context of e-government. Also, as future work, it is possible to carry out complementary evaluations using other methods such as reviews by specialists and user testing. Also, inspections can be carried out in other contexts and countries in order to determine the degree of awareness about the importance of accessibility in the development of software products, as well as to identify the gap betw- een the degree of accessibility that is offered by websites and what is actually provided by them, given the fact that there are national and international standards.

ACKNOWLEDGMENT

This research was developed by the HCI, Design, User Experience, Acces- sibility and Innovation Technology Research Group (HCI-DUXAIT). HCI- DUXAIT is a research group of PUCP. We thank all the specialists of the research group who collaborated in the conduction of this work.

REFERENCES

Benites Alfaro, F. D., & Zapata Del Río, C. M. D. P. (2018). Peruvian Public Universities and the Accessibility of Their Websites. In W. Marcus Aaron and Wang (Ed.), Design, User Experience, and Usability: Users, Contexts and Case Studies (pp. 589–607). Springer International Publishing. https://doi.org/10.1007/978-3- 319-91806-8_46 Centers for Disease Control and Prevention (CDC). (2020). Disability and Health

Promotion. https://www.cdc.gov/ncbddd/disabilityandhealth/infographic-disabil ity-impacts-all.html

A Comparison of Accessibility Assessment Tools 101

Instituto Nacional de Estadística e Informática - Perú (INEI). (2021). Perú: Estadísticas de las personas con alguna discapacidad. https://conadisperu.gob.pe/observa torio/estadisticas/peru-estadisticas-de-las-personas-con-alguna-discapacidad/ Ministerio de Salud - Perú. (2021). Sala Situacional COVID-19 Perú. https://covid19.

minsa.gob.pe/sala_situacional.asp Nuñez, A., Moquillaza, A., & Paz F. (2019). Web Accessibility Evaluation Methods: A Systematic Review. In W. Marcus Aaron and Wang (Ed.), Design, User Experie- nce, and Usability. Practice and Case Studies (pp. 226–237). Springer International Publishing. https://doi.org/10.1007/978-3-030-23535-2_17 Paz, F., Paz, F. A., Villanueva, D., Moquillaza, A., & Arenas, J. (2020). Acces-

sibility Evaluation of Peruvian E-Commerce Websites. International Journal of Advanced Trends in Computer Science and Engineering, 9(1), 558–561. https://doi.org/10.30534/ijatcse/2020/76912020 Paz, F., & Paz, F. A. (2021). Relation Between Usability and Accessibility: A Case Study in Peruvian E-Commerce Websites. In R. and G. F. Ahram Tareq and Taiar (Ed.), Human Interaction, Emerging Technologies and Future Applications IV (pp. 384–390). Springer International Publishing. https://doi.org/10.1007/978-3-030- 74009-2_49 Paz, F., Paz, F. A., Moquillaza, A., Collantes, L., Arenas, J. J., & Villanueva, D. (2021).

Comparison of Accessibility Tools for the Evaluation of E-Commerce Websites. In T. and K. W. and D. B. G. and T. R. Russo Dario and Ahram (Ed.), Intel- ligent Human Systems Integration 2021 (pp. 876–881). Springer International Publishing. https://doi.org/10.1007/978-3-030-68017-6_130 Petrie, H., & Kheir, O. (2007). The Relationship between Accessibility and Usability of Websites. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 397–406. https://doi.org/10.1145/1240624.1240688 United Nations (UN). (2006). Article 9 – Accessibility. https://www.un.org/develop

ment/desa/disabilities/convention-on-the-rights-of-persons-with-disabilities/artic le-9-accessibility.html World Health Organization (WHO). (2021). Disability and health. https://www.wh o.int/en/news-room/fact-sheets/detail/disability-and-health World Wide Web Consortium (W3C). (2014). Website Accessibility Conformance

Evaluation Methodology (WCAG-EM) 1.0. https://www.w3.org/TR/WCAG-EM/ World Wide Web Consortium (W3C). (2020). Web Accessibility Evaluation Tools List. https://www.w3.org/WAI/ER/tools/

Cover of the proceedings volume Human Factors in Accessibility and Assistive Technology

Explore the complete digital volume

Human Factors in Accessibility and Assistive Technology

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-13-1

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →