Lorelie C. Grepo, Benette P. Custodio (2020). A Comparison of the Usability of Locally-Produced and Commercially-Acquired Telemedicine Device for Filipino Health Workers. In: Advances in The Ergonomics in Manufacturing: Managing the Enterprise of the Future. AHFE Open Access, vol. 12. https://doi.org/10.54941/ahfe100439
With the advancement of technology today, various industries have benefitted from innovation of systems that have been and are being developed, with the healthcare sector being one of them. Usability evaluation of medical devices is essential to ensure its effectiveness and to increase acceptability of the technology. In the Philippines, there has been a project on development of a locally-produced medical device. Similar to any other health care applications, there is a need to ensure that this is usable for Filipino health workers. A usability evaluation was conducted on the prototype of this telemedicine device, focusing on maternal monitoring, with user testing and interviews. Another similar device (commercially-acquired) was evaluated and compared to the existing design. Results of the evaluation indicate that there were usability problems in terms of the hardware and interface for both the locally-produced and commercially-acquired device designs. The former has a lot to improve in terms of its interface. Both still needs further improvement to make it easier for users to use the device. Usability evaluation has been very useful in identifying the areas for improvement in the design of the medical device and is seen as an essential part of its development.
Medical devices play a vital role in health care systems. Innovations on this filed are intended to facilitate a more efficient and more effective delivery of patient health care. Today, a vast number of medical equipment, computer applications and medical systems have been and are being developed. To ensure that the maximum benefit is achieved from these, it is essential that these systems be usable for its intended users. Usability of systems indirectly affects user acceptance (Maguire, 2001) and thus, developers must be able to come up with innovative and good system design.
According to Jaspers (2009), poor design often becomes a hindrance to user adoption and utilization of interactive health care applications. It makes the system difficult to use and makes it hard for the users to learn them. In this research, other studies were discussed highlighting the effects of poor design to errors (Jaspers, 2009; Kushniruk, et. al., 2005; Han, et. al. 2005; Ash, et. al., 2004; Campell, et. al., 2006; Horsky, et. al., 2005; Peute, et. al., 2007; Ergonomics in Manufacturing (2020)
Koppel, et. al., 2005).
In the Philippines, the government has started projects on development of local devices and computer applications intended to provide better service to its people, one of which is a telemedicine device. This is a medical device developed with capability of measuring patient vital signs and other data through sensors and at the same time, a capability for telemetry. The main function of this device is focused on maternal health care – check up and monitoring. These devices are intended to be deployed to many provinces and islands of the country to improve health services especially in the remote areas. Currently in its development phase, a usability evaluation is a necessity to know how it is going to be used by the Filipino health workers especially in the rural areas. As Hyman (1994) and Obradovich and Woods (1996) discussed, the risks of having human errors in medical equipment use increases if the machine interface is poorly designed, thus, the need to evaluate this locally-produced equipment. During the initial deployment of the project, another version of the device, commercially-acquired, was distributed in some areas. To be able to help in identifying the improvements needed in the development, the usability of this commercially-acquired device was conducted and compared to the one being developed to know how it will fit the Filipinos.
USABILITY EVALUATION Medical device usability has become a growing interest (Mchome, et. al. 2010). This is rationalized not just for the criticality of these systems but also for better product development and financial benefits.
Various usability evaluation techniques exist today. In general, these can be classified into two: user-based or expert-based (Jordan, 1998). In terms of deciding which technique will be best for different product development phases, Tan, et. al. (2009) recommends that heuristic evaluation be utilized at the early stages and then conduct user testing at the latter stages of the development. They also discussed some of the research works covering usability evaluation methods, where each is applicable and the strong points of each technique. They compared heuristic evaluation, the most commonly used expert-based technique, to the most commonly used user-based technique, user testing. Heuristic evaluation permits the possibility of identifying what is possible rather than just to what already exists. Jeffries et. al. (1991) also compared user testing and heuristic evaluation and came up with the conclusion that the former was able to identify more problems but the more severe ones were identified through user testing. Liljegren (2006) compared hierarchical task analysis, cognitive walkthrough, heuristic evaluation and user testing with the following criteria: thoroughness, validity, reliability, cost effectiveness and clarity. Liljegren (2006) and Lindgaard (2006) both recommends user testing to be used as primary technique in usability evaluation since it addresses the “difficult to make errors” aspect. For Choi, et. al. (2010) and Stinson (2010), user testing is the preferred methodology.
For medical devices, a number of studies have also been done for usability evaluation. Jaspers (2009) has compared heuristic evaluation, cognitive walkthrough and think aloud method for a specific medical device and has concluded that each of the technique has its own advantages and disadvantages and further recommended that a combination of the techniques will be more powerful than any individual method.
Although the most commonly used and lowest-cost option is an expert-based heuristic evaluation, there are cases in which this is not viable because of the lack of experts in the field. This applies in the current setup of this telemedicine device development. Given the constraint on lack of experts and the primary need of the developers at this phase, which is to determine the need of the users and how the device is going to be used by medical personnel in different areas, the evaluation conducted is mostly user-based techniques: user testing, questionnaires and interviews.
Methodology
The usability evaluation in this study was conducted during the initial deployment of the devices to selected areas (combination of rural and urban) within the Philippines. Trainings, which include use of health information system and use of the telemedicine device, were conducted on the different health units. Each health unit is deployed only one version of the device, either the locally-produced or the commercially-acquired one. Both of these, however, were designed and customized to work with the current health information system being developed and implemented as well. The main functionalities that the devices have are blood pressure, pulse oxymeter, tocometer, fetal heart rate monitor and ECG.
Before data was gathered, the medical workers were given orientation and were taught about the different functions of the device. After the training, the device is used in actual patients and this is when the data for usability was collected. Videos were taken while the users interact with the device for checkup and monitoring of patients. These were later analyzed for identification of the problems encountered while it is being used. These were also used to obtain an estimate of how long the health workers take in doing the tasks discussed in the next section. After the testing, specific users were interviewed to elaborate the problems they encountered or have difficulties doing in the device. Then analysis of the data was done and the devices were compared in terms of its usability for Filipino health workers.
Test Subjects McClelland (1995) emphasizes that a representative user population must be chosen as sample users for the trial. With this, the test subjects in the study were selected from the participants in the training conducted for the Telemedicine program, part of which is on using the medical device being developed. The participants in the training consists of Municipal Health Officer(s), Public Health Nurses, Rural Health Midwives and Doctors, the primary users of the device. Their ages range from 22 to 60 years old, with an average of 42. It is noticeable that the variability between ages is high but since the study is limited by the training and the personnel in different health units, everyone who tested the device was included in the study.
Garner et.al. (2002) highlights the importance of including novices in usability tests since they are the ones who encounter the most serious and most number of errors. In this research, most of the users are classified as novices although there are a few who can be considered experienced already in using the device as some health centers or hospitals are using or have used a similar equipment.
For Virzi (1992) and Lewis (1994), 18 people are needed to identify 90% of the usability problems. Hwang & Salvendy (2010) on various heuristic evaluation studies done before found out that the 10±2 rule would apply. According to Ericsson and Simon (1993), 8 subjects suffice to gain a thorough understanding of task behavior or to identify usability problems with a computer system (Nielsen, 1994). Other usability specialists recommend 5 participants to achieve the maximum benefit-cost ratio (Nielsen & Landauer, 1993) and can identify about 80% of the usability problems. For user testing, Nielsen (1999) recommends observation of about 10 subjects for detection of major usability problems in an interface. According to him, having subjects beyond 5 to 6 will not economically justify the tradeoff between additional cost and additional problems that will be identified. Given that the usability evaluation during this time is limited by the training in sites where the devices were deployed, the number of subjects varied per site but ranges from 2 to 6 per health unit. A total of 4 health centers were included.
Tasks The tasks observed during the evaluation of the medical device being studied are described below.
1. Patient record is created and/or accessed from the health information system. 2. Medical device is set-up. The sensors are attached to the device. The sensors are then attached to the patient to obtain the mother’s vital signs (blood pressure, pulse, oxygen saturation, contractions excluding ECG) and the fetal heart rate. It is important to note here that the main use of the device observed during this time was for pre-natal check-up and maternal monitoring. 3. After vital signs have been measured, data obtained is exported and saved to the information system under the patient record.
ANALYSIS AND DISCUSSION OF RESULTS The average task duration of each activity is given below.
Table 1: Duration of tasks (in minutes) in using the devices Tasks Locally-developed Commerciallyacquired
Comparing the times given in Table 1, it can be noticed that there is no much difference in the duration for the first two tasks. The general flow of activities in the patient record creation and/or accessing it (if already existing) is almost the same for the two devices mainly because it uses the same hospital information system linked to it. The commercially-acquired device may take a little bit longer because it requires another application to be opened to link the device to the information system and vice versa.
For device setup and getting vital signs, the difference is more negligible for the two devices. This can be attributed to the number of sensors and how the sensors are to be used in the patients. The same set of sensors are provided for both devices – blood pressure, pulse oxymeter, tocometer and fetal heart rate monitor. The ECG sensor was not observed to be used during the training. The time for the last task, saving the data reflected in the device into the hospital information system resulted in a significant difference between the two devices. This is because the commercially-acquired device needs an extra set of steps to be saved in the system – the user has to extract the data from the device through another application and then this application will be linked to the information system (device – application – patient record).
The qualitative part of the evaluation mainly came from observations on how the users interacted with the device and based on interviews conducted by the proponents. Tables 2 and 3 below compares the different aspects of the device design which were seen to affect the users while performing the tasks assigned to them from the videos taken.
Table 2: Hardware-related usability issues identified in using the devices Aspect Locally-developed Commercially-acquired
It can be seen from Table 2 that each version of the device has its own advantages and disadvantages in terms of its design.
Starting with the keyboard or input device, the commercially-acquired device has provided shortcuts (buttons) in the device itself which makes it easier for the health workers to start an action (e.g. BP) in one step. It was noticed during the experiment that older participants had hard time adjusting to a touch-screen type keyboard as opposed to the normal computer/laptop keyboard.
For the components part, the locally-produced device is advantageous because it has less number of cables/sockets to attach. However, both medical devices require a certain orientation on how the cables/sockets must be placed which makes takes more time for the workers. Another factor that caused confusion on the users is the color-coding system used in the cords. The users tend to match the color codes to the wires itself and not on the ring around the socket which resulted to wrong cable connected to the port.
The commercially-acquired device has a built-in printer that prints the mother’s contractions and baby’s heart rate real-time as the readings are being taken by the machine. This is very quickly done. Usually, a printout of this type is needed when the patients have to be referred to a hospital for giving birth. This is opposed to the locally- manufactured device that, although it has the capability to generate the graphs/plot similar to what the commercially-available prints, to obtain a printout, the user has to access a separate computer connected to a printer and retrieve the patient’s file from it before printing. This takes more tasks and time to finish.
Lastly, placement is one advantage of the locally-produced device since it requires less space in the clinic. Only the telemedicine device and its components need a space in the labor room as opposed to the commercially-acquired one which has to be connected to a laptop or computer for the data to be retrieved from the device. For health centers with space limitations, this is a disadvantage.
Table 3: Interface-related usability issues identified in using the devices Aspect Locally-developed Commercially-acquired
With regard to the interface, it can be seen in Table 3 that the commercially-acquired device has an advantage compared to the locally-developed version. The aspects pointed out in the table are the areas identified where the design of the locally-produced device must be improved. Since the medical device is still being developed, the issues identified here serves as input to the developers to be incorporated in the next release of the telemedicine device.
In terms of the content, most of the errors or messages in the locally-produced device need to be re-stated to be clearer and at the same time easier to understand for the users. In particular, there was a status message indicating “CRITICAL” so the health workers thought that the patient is in critical condition but the system meant “POOR” for the signal that the device is getting. With this, it can create possible confusion on the users. It is suggested that messages and warnings be made clear so that misperception can be avoided during use.
Consistency of the interface was also pointed out as an area for improvement because in the different vital signs where the readings can be seen, different layout and arrangement of keys and buttons were seen. For example, there were windows where the “OKAY” button is on the upper right, in some cases it is on the bottom of the window. Given this inconsistency can also contribute to user confusion in using the system and furthermore, make the tasks longer because locating the needed keys/buttons may still take time of the health workers.
For the system status, however, both interfaces of the commercially-acquired and locally-manufactured device, can be improved by sending a prompt to the user when the device finishes getting measurements from the patient, for example, blood pressure, through a pop-up message in the screen or though a sound. This was recommended since it was noticed that when the health workers get vital signs, they wait for the device to finish but are unsure of how long they should wait and they are uncertain of when to say the measurements have already stabilized.
Lastly for the screen, since both devices have limited screens, many users requested for larger fonts and larger images so that they could easily read and see the vital signs. In the commercially-acquired device, all the vital signs are displayed at the same time so the health workers just need to glance at the different parts of the screen to obtain the readings desired. This is compared to the locally-manufactured device, which shows the vital signs one at a time. During monitoring, this is a disadvantage because the health workers will have to shift from one window to another to get a “complete picture” of the patient’s condition.
One disadvantage of heuristic evaluation is that sometimes, the real needs of the users are not identified because of the lack of experts with enough knowledge on the discipline and technology being studied (Jaspers, 2008). In this research, other qualitative issues were identified from interviews with the health workers themselves. The insights of the nurses, midwives and doctors on the use of the device were obtained and examined. The following items summarizes these results:
1. Having the device in their health centers will help improve the efficiency especially in pre-natal checkup and patient monitoring. Respondents are happy with the device given to them but some are hesitant to use the system as part of their routine because they find the management of online patient records too tedious and time-consuming. Doing the tasks manually is significantly faster for them (approximately 5 minutes versus 8 to 10 minutes using the devices).
2. Midwives aged 40’s to 50’s admitted that they would find a hard time using the device (both versions) since it is already difficult for them to learn new technology.
3. There were components of the device in which some health workers are not familiar to, especially the ones in the rural areas. The subjects specifically requested that it should be part of the training for them, for example, interpreting the graphs generated for contractions and fetal heart rate. Ergonomics in Manufacturing (2020)
4. Doctors gave their suggested layout on how the locally-produced interface can be improved based on their experiences with other devices similar to it. In almost all the sites where the locally-produced device was deployed, the doctors emphasized the need for displaying the vital signs all at the same time (one screen) for labor monitoring.
5. Use of the device depends on the number of patients that the health center caters to. In a clinic where there is a significantly large number of mothers giving birth, the health workers are not sure if they could actually use the device. They suggested that it will be more appropriate for pre-natal checkup than that of lying-in monitoring since they only have one set of the device and there are a lot of patients that would need attention. Identifying to whom the device must be used will be another issue for them.
After the quantitative and qualitative evaluations, it can be seen that a lot of issues related to the use of the device were identified. These, however, will serve as opportunities for improvement in the current design of the locally- manufactured device.
CONCLUSIONS
The user-testing and interviews conducted from the evaluation of the medical devices led to identification of usability issues. Comparing the locally-manufactured and commercially-acquired devices highlighted the advantages and disadvantages of each. In terms of the interface, the locally-produced device still has a lot to improve in terms of its design. The main benefit from this study is that the developers of the local medical device will be able to identify the specific aspects of system that needs to be enhanced.
Since the usability evaluation conducted here focused during the initial deployment of the device (prototype stage), it is recommended that after the initial suggestions on design improvement is implemented, another usability evaluation be conducted to check for other problems that users may encounter in using the device.
ACKNOWLEDGMENTS
The authors would like to thank Dr. Luis G. Sison, the RxBox Development Team and the Electrical and Electronics Engineering Institute of the University of the Philippines Diliman, the National Telehealth Center of University of the Philippines Manila, the Department of Science and Technology and Dr. Aura C. Matias and the Human Factors and Ergonomics Laboratory of the Industrial Engineering and Operations Research Department of University of the Philippines Diliman for all their help, participation and contribution to this research.
REFERENCES
Äijö, R. & Mantere, J. (2001) “Are non-expert usability evaluations valuable?”, 18th International Symposium on Human Factors in Telecommunication Proceedings Ash, J., Coiera, A., Berg, M. (2004) “Some unintended consequences of information technology in health care: the nature of
patient care information systems”, Journal of American Medical Informatics Association 11, pp. 104-112 Bond, R.R., Finlay, D.D., Nugent, C.D., Moore, G., Guldenring, D. (2014), “A usability evaluation of medical software at an expert conference setting”, Computer Methods and Programs in Biomedicine 113, pp. 383-395 Bonnie, J. (1996), “Evaluating Usability Evaluation Techniques”, ACM Computing Surveys (CSUR) – Special issue: position
statements on strategic directions in computing research, Vol. 28, Issue 4es Bright, T.J., Bakken, S., Johnson, S.B. (2006), “Heuristic evaluation of eNote: an electronic notes system”, Proceedings of AMIA, p. 864 Brooke, J. (1996), “SUS: a "quick and dirty" usability scale”, In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L.
McClelland (Eds.), Usability Evaluation in Industry, London: Taylor and Francis, pp. 189-194
Campbell, E.M., Sittig, D.F., Ash, J.S., Guappone, K.P., Dykstra, R.H. (2006) “Types of unintended consequences related to computerized provider order entry”, Journal of American Medical Informatics Association 13 (5), pp. 547-556 Chevalier, A., Bonnardel, N. (2007), “Articulation of web site design constraints: Effects of the task and designer’s expertise”,
Computers in Human Behavior, vol. 23, pp. 2455-2472 Choi, J., Bakken, S. (2010), “Web-based education for low-literate parents in Neonatal Intensive Care Unit: development of a website and heuristic evaluation and usability testing”, International Journal of Medical Informatics, pp. 565-575 De Jong, M. & Lentz, L. (2006), “Scenario evaluation of municipal Web sites: Development and use of an expert-focused
evaluation tool”, Government Information Quarterly Vol 23, pp. 191-206 Ericsson, K.A. & Simon, H.A., (1993), “Protocol Analysis: Verbal Reports as Data”, MIT Press, Cambridge Garmer, K., Liljegren, E., Osvalder, A., Dahlman, S. (2002), “Application of usability testing to the development of medical equipment. Usability testing of a frequently used infusion pump and a new user interface for an infusion pump developed with a new Human Factors approach”, International Journal of Industrial Ergonomics 29, pp. 145-159 Han, Y.Y., Carcillo, J.A., Venkataraman, S.T., Clark, R.S., Watson, R.S., Nguyen, T.C., Bayir, H., Orr, R.A. (2005),
“Unexpected increased mortality after implementation of a commercially sold physician order entry system”, Pediatrics 16 (6), pp. 1506-1512 Hollingsed, T. & Novick, D. (2007), “Usability Inspection Methods after 15 Years of Research and Practice”, SIGDOC '07 Proceedings of the 25th annual ACM international conference on Design of communication, pp. 249-255 Horsky, J., Kuperman, G.J., Patel, V.L. (2005), “Comprehensive analysis of a medication dosing error related to CPOE”,
Journal of American Medical Informatics Association 12 (4), pp. 377-382 Hwang, W. & Salvendy, G. (2010), “Number of People Required for Usability Evaluation: The 10±2 Rule”, Communications of the ACM Vol. 53, No. 5, pp. 120-133 Hyman, W.A. (1994), “Errors in the use of medical equipment”, in Bogner, M.S. (Ed.), Human Error in Medicine. Lawrence
Erlbaum Associates, Inc. New Jersey, pp. 327-347 ISO/IEC, IEC 62366: 2007 Medical Devices – Application of Usability Engineering to Medical Devices, 2010. Available: http://www.iso.org/iso/catalogue detail.htm?csnumber= 3859 (01.11.10). Jaspers, M.W.M. (2009), “A comparison of usability methods for testing interactive health technologies: Methodological aspects
and empirical evidence”, International Journal of Medical Informatics 78, pp. 340-353 Jaspers, M.W.M., Steen, T., Bos van den, C., Geenen, M. (2004), “The think aloud method: a guide to user interface design”, International Journal of Medical Informatics 73 (11-12), pp. 781-795 Jeffries, R. & Desurvire, H. (1992), “Usability Testing vs. Heuristic Evaluation: Was there a contest?”, SIGCHI Bulletin, Vol.
24 No. 4, pp. 39-41 Jeffries, R., Miller, J., Wharton, C., Uyeda, K. (1991), “User interface evaluation in the real world: a comparison of four techniques”, Proceedings ACM CHI’91 Conference, New Orleans, LA, pp. 119–124 Johnson, T.R., Tang, X., Graham, M.J., Brixey, J., Turley, J.P., Zhang, J., Keselman, A., Patel, V.L. (2007), “Attitudes toward
medical device use errors and the prevention of adverse events”, Jt. Comm. J. Qual. Patient Saf. 33, pp. 689-694 Jordan, P. (c1998), “An Introduction to Usability”, London: Taylor & Francis Jordan P., Thomas B., Weerdmeester, B. & McClelland, I. (c1996), “Usability Evaluation in Industry”, Taylor and Francis Koppel, R., Metlay, J.P., Cohen, A., Abaluck, B., Localio, A.R., Kimmel, S.E., Strom, B.L. (2005), “Role of computerized physician order entry system in facilitating medical errors”, Journal of American Medical Association 293 (10), pp. 1197- 1203 Kushniruk, A.W., Triola, M.M., Borycki, E.M., Stein, B., Kannry, J.L. (2005), “Technology induced error and usability: the
relationship between usability problems and prescription errors when using a handheld application”, International Journal of Medical Informatics 74, pp. 519-526 Lathan, C.E., Sebrechts, M.M., Newman, D.J., Doarn, C.R. (1999), “Heuristic evaluation of a web-based interface for telemedicine”, Telemed. J. 5 (2), pp. 177-185 Lewis, C. (1982), “Using the “think-aloud” method in cognitive interface research”, Research Report RC9265, IBM T.J.
Watson Research Centre, Yorktown Heights, NY. Lewis, R.J. (1994), “Sample sizes for usability studies. Additional considerations”, Human Factors 36 (4), pp. 368-378 Ling, C. & Salvendy, G. (2005), “Extension of heuristic evaluation method: a review and reappraisal”, Ergonomia IJE&HF, Vol. 27 No. 3, pp. 179-197 Maguire, M. (2001), “Methods to support human-centered design”, International Journal of Human-Computer Studies, pp. 587-
634 Mchome, S., Sachdeva, S., Bhalla, S. (2010), “A brief survey: usability in healthcare”, International Conference on Electronics and Information Engineering 1, pp. 463-467 McClelland, I. (1995), “Product assessment and user trials”, in Wilson, J.R., Corlett, E.N. (Eds.), Evaluation of Human Work.
Taylor and Francis Ltd., London, pp. 118-247 Molich, R., Nielsen, J. (1990), “Improving human-computer dialogue”, Communications of the ACM 33(3), pp. 338-348 Nielsen, J. (c1999), “Usability Engineering”, Boston: Academic Press Nielsen, J. (1994), “Estimating the number of subjects needed for a think aloud test”, International Journal on Human-Computer Studies 41, pp. 385-397 Nielsen, J. (1994), “Guerilla HCI: Using discount usability engineering to penetrate intimidation barrier”, in R.G. Bias, D.J.
Mayhew (Eds.), Cost-justifying Usability, Academic, Cambridge, MA Nielsen, J. (1992), “Finding usability problems through heuristic evaluation”, in P. Bauersfeld, J. Bennet, G. Lynch (Eds.),
Proceedings of the ACM CHI 1992 International Conference on Human Factors in Computing Systems, ACM, New York, 1992, pp.277-278 Nielsen, J., Landauer, T.K. (1993), “A mathematical model of the finding of usability problems”, Proceedings of the 1993 Conference on Human Factors in Computing Systems, pp. 206-213 Nielsen, J., Mack, R. (1994), “Usability Inspection Methods”, Wiley, New York 36 (12), pp. 1069-1074 Nielsen, J., Molich, R. (1990), “Heuristic Evaluation of User Interfaces”, Proceedings of the ACM Conference on Human
Factors in Computing, pp. 249-256 Nikolaos, T., Nikolaos, A. & Vassilis, K. (2008), “The effective combination of hybrid usability methods in evaluating evaluating educational application of ICT: Issues and Challenges”, Education Information Technology Vol. 13, pp.55-76 Obradovich, J.H., Woods, D.D. (1996), “Users as designers: how people cope with poor HCI design in computer-based medical
devices”, Human Factors 38, pp. 574-592 Oliver, M. (2000), “An introduction to the evaluation of learning technology”, Educational Technology Research & Development Vol. 3 (4), pp. 20-30 Peute, L.W. & Jaspers, M.W.M. (2007), “The significance of usability evaluation of an emerging laboratory entry system”,
International Journal of Medical Informatics 76 (2-3), pp. 157-168 Polson, P.G. & Lewis, C.H. (1990), “Theory-based design for easily learned interfaces”, Human-Computer Interactions 5, pp.
191-220 Polson, P, Lewis, C. Rieman, J., Wharton C. (1992), “Cognitive walkthroughs: a method for theory-based evaluation of user interfaces”, International Journal of Human-Computer Studies 36, pp. 741-773 Sears, A. (1997), “Heuristic walkthroughs: finding the problems without the noise”, International Journal on Human-Computer
Interactions 9 (3), pp/ 213-234 Seffah, A, Donyaee, M., Kline, R. & Padda, H. (2006), “Usability measurement and metrics: A consolidated model”, Software Quality Journal Vol. 14, pp. 159-178 Squires, D. & Preece, J. (1992), “Predicting quality in educational software: Evaluating for learning usability and the synergy
between them”, Interacting with Computers Vol. 11, pp.467-483 Ssemugabi, S. & de Villiers, R. (2007), “A Comparative Study of Two Usability Evaluation Methods Using a Web-based E- Learning Application”, South African Institute of Computer Scientists and Information Technologies Proceedings, pp. 132- 142 Stinson, J. (2010), “Usability testing of an online self-management program for adolescents with juvenile idiopathic arthritis”,
Journal of Med. Internet Res. 12, e30 Tan, W., Liu, D., Bishu, R. (2009), “Web Evaluation: Heuristic evaluation vs. user testing”, International Journal of Industrial Ergonomics, vol. 39, pp 621-627 Tang, Z., Johnson, T.R., Tindall, R.D., Zhang, J (2006), “Applying heuristic evaluation to improve the usability of a telemedicine
system”, Telemed. J. E. Health 12 (1), pp. 24-34 Tullis, T. & Albert, B. (c2008), “Measuring the user experience: collecting, analyzing, and presenting usability metrics”, Amsterdam; Boston: Elsevier/Morgan Kaufmann Virzi, R.A. (1992), “Refining the test phase of usability evaluation. How many subjects is enough?”, Human Factors 34 (4), pp.
457-468
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in The Ergonomics in Manufacturing: Managing the Enterprise of the Future
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2103-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional