Tore J Larsson, Hedvig Aminoff, Mannan Mridha (2019). A Consumers’ Testing Approach to the Usability of Medical Technology - Insulin Pumps and CGM Systems. In: Advances in Safety Management and Human Factors. AHFE Open Access, vol. 5. https://doi.org/10.54941/ahfe100156
Five different insulin pumps and three systems for continuous glucose monitoring were subjected to usability tests at the School of Technology and Health. Each pump was trialed and rated by 30 respondents; 20 students with no experience of diabetes and 10 diabetic pump users. Each of the CGM systems was trialed and rated by 10 non-diabetic students. All participating students were enrolled in Medical Technology (Royal Institute of Technology) or Occupational Therapy (Karolinska Institute). The technical performance of pumps and CGM systems was tested independently. The respondents handled the insulin container, the software, the buttons, the screen and the manual through five scenario-based tasks. The trials and the accompanying attitude items were based on the ISO definition of usability. Efficiency was measured as the proportion of respondents succeeding to perform the tasks in less than 15 minutes, combined with the average time to do so. Effectiveness was the quotient of success frequency over average performance time. Satisfaction was the average distribution on the attitude items related to software, screen, buttons and manual. All products were ranked against each other within each separate test and the rank scores accumulated. There were significant differences in the scoring of the individual insulin pumps and CGM systems.
The Dental and Pharmaceutical Benefits Agency, TLV, is a Swedish central government Agency whose remit is to determine if a pharmaceutical product or dental care procedure shall be subsidized by the state. This Agency has commissioned the School of Technology and Health to develop and apply a standardized evaluation model for the usability of medical technology products. Specifically, five different insulin pumps and three systems for continuous glucose monitoring available on the Swedish market were chosen for the testing.
METHOD
An evaluation model is suggested, which is simple and aimed at the ordinary consumer, defining the medical technology under scrutiny as a piece of mainstream consumer goods. The model is based on the ISO-definition of usability, where efficiency, effectiveness and satisfaction are measured (ISO/IEC 25062:2006; FDA 2006; NHS 2010; Jordan 1998). A number of scenario-based tasks are performed on the product by a group of respondents, who are also giving value judgements on different aspects of the product. According to this model efficiency was defined Safety Management (2019) as the proportion of respondents who managed to finish the task with or without assistance, within 15 minutes, combined with the average time to solve the task. Effectiveness was defined as the quotient of success frequency over average performance time. Satisfaction was defined as the average distribution on the attitude items related to software, screen, buttons and manual. In addition to this, the ten queries of the System Usability Scale (Bangor et al 2008; Lewis and Sauro 2009; Sauro 2011) were put to the respondents. Products and respondents
The five insulin pumps included in the tests were Paradigm Veo (Medtronic), Animas Vibe (Rubin Medical), Dana R (Nordic Infu Care), Accu-Chek Combo (Roche Diagnostics Scandinavia) and Omnipod (Ypsomed). The three glucose-monitoring systems were Mini Link (Medtronic), Dexcom G4 Platinum (Rubin Medical) and Freestyle Navigator II (Abbott Diabetes Care).
Each of the five pumps was tested by 20 university students (novices) and 10 diabetes patients (users). Each of the glucose monitoring systems was tested by 10 university students (novices). The students were recruited at the Royal Institute of Technology and the Karolinska Institute and were undergraduates in medical technology or occupational therapy with no prior knowledge or experience of diabetes. The diabetes patients were recruited from the large hospitals in Stockholm and Uppsala, and they were everyday users of the specific pump they were asked to test. The Omnipod pump was not available in the market; randomly selected users of other pumps tested it. The tests were conducted individually and took place in rooms at the University or the Hospital. All the pumps and CGMs were subjected to precision tests. Four different flow rates in the pumps were checked over time and the precision of the three CGM systems was compared to traditional glucose measurement.
Tasks The respondents were given 15 minutes to study the manual and they were encouraged to set the correct time and date on the product. With the five pumps, the respondents were asked to insert the insulin capsule and prime the pump, to set a basic 24-hour program of delivery, to set a temporary 24-hour program, to set an alternative 24-hour program, and to deliver an immediate dose of insulin (bolus). With the three CGM systems the respondents were asked to set the value span for normal glucose levels, set the alarm function for highest and lowest values, and set the snooze function to repeat alarm after a certain period of time.
RESULTS
Efficiency, pumps Efficiency was measured as frequency of success and average time to conclude each task in each pump.
The users scored the Medtronic pump similar to the novices; low success rate but the fastest time. The Omnipod had a high frequency of success, in spite of the fact that this was a new, unknown product to the user.
Table 4: Time pumps, users (n=10)
Effectiveness, pumps Effectiveness was recorded as completion rate/average time to finish the task. A high number represents a high degree of effectiveness.
Performance times were shortest with the Navigator system. Completion rate efficiency is a direct mirror of performance time since completion rate was 100% for all tasks and all systems. A high number represents high effectiveness.
Table 8: Effectiveness CGM, novices (n=10)
Satisfaction and usability, pumps A summed average of the responses to the six attitude items - on the pump, the program, the screen, the buttons, the manual and the insulin container – can show the proportion of respondents judging the respective aspects of the product as negative – “complicated”. Accordingly, a low number is positive.
Table 9: Satisfaction, novices (Nov) and users (Use) (n=30)
The Nov Use Nov Use Nov Use Nov Use Nov Use
Animas pump had the average best scores of satisfaction as measured by novices and users combined.
The System Usability Scale (SUS, Brooke 1996) is a survey with 10 items, which represents a combined measure of users’ subjective assessment of the usability of a product or system. The result is a number between 0-100. The scale is viewed as having good validity and reliability (Bangor et al 2008). The average SUS value is 68 and this can be seen as a benchmark; a SUS value above 68 indicates better usability than the average (Sauro 2011).
Table 10: System Usability Scale, pumps (n=30) Novices SUS Accu-chek 46 Animas 71 Dana 60 Medtronic 64 Omnipod 61
The highest score on usability according to SUS was recorded for the Animas pump by novices (71) and by users (77). The Accu-Chek pump attracted the lowest score from novices (46) and the Omnipod pump the lowest score from users (58).
Satisfaction and usability, CGM systems The assessment of the CGM systems – on the five aspects of simplicity, software, screen, buttons and the manual – was averaged and ranked. Dexcom was ranked the highest, the two other systems received the same rank order.
Table 11: Satisfaction, CGM novices (n=10)Table 12: System Usability Scale, CGM systems novices (n=10) CGM System Usability Scale (SUS) Rank
Precision, pumps The pumps investigated were technically similar and fulfilled the specifications given in the manuals. The pumps were tested four different flow rates - 9.9, 3.3, 1.1 and 0.1 units/hour. Measurements were undertaken with the help of a high-precision scales (Sartorius BD301S) and readings were with a precision of 0.1 mg fluid. Insulin was substituted with sodium chloride (Braun 9mg/ml) and time was measured with a precision of + 1 min.
The measurements on flow rates 9.9, 3.3 and 1.1 units/hour showed a variation of less than 5% in all of the five pumps. However, measurements of the very low flow rate 0.1 units/hour showed that three of the pumps – Animas, Dana and Omnipod – delivered in excess of the setting.
Table 13: Tested flow rates, pumps
Precision, CGM systems
The three CGM systems, applied to a healthy member of the research team, registered measurement values over 72 hours within the target area (3.7 mmol/l to 7.8 mmol/l). The systems measured with equal precision and in agreement with glucose values taken from blood samples every 12 hours.
Estimated price pumps The 100 novice respondents with no experience of diabetes and insulin pumps were asked to estimate the retail price of the pump they were testing. Two students responded with a very high value, the other 98 estimated the price at less than SEK 10 000. On average, the estimate is SEK 4 500:- . The Omnipod was not on the market in 2012.
Table 14: Estimated and actual retail prices insulin pumps 2012.
SUMMARY OF RESULTS The tests undertaken can be used to express four aspects of the comparative utility value of the product: efficiency, effectiveness, satisfaction and usability. Putting all the measurements together and averaging the rank orders of each of the products sums up the usability test:
Table 15: Sum of usability test, pumps (N=150)
The overall highest ranked is Animas, followed by Medtronic. The lowest ranked is Accu-Chek and Dana. The new product Omnipod is ranked in the middle.
Novices and Users differ somewhat in their judgement and performance in the tests of the pumps; Novices give Accu-Chek the lowest score, Users give the lowest score to Omnipod.
best efficiency - Animas, Medtronic and Omnipod.
best effectiveness - Medtronic.
best satisfaction - Animas and Medtronic.
best usability (SUS) - Animas.
The results are in line with the conclusions from an earlier design comparison of Animas’ and Medtronics’ pumps conducted at the University of Michigan (Best et al 2011).
Table 16: Sum of usability test, CGM systems, novices (N=30) CGM Efficiency Effectiveness Satisfaction Usability (SUS) Total
The overall highest ranked is the Dexcom system, followed by Navigator and Guardian.
DISCUSSION AND CONCLUSIONS This evaluation of insulin pumps and CGM systems shows that there are quite measurable differences in usability, operationalized as efficiency, effectiveness and satisfaction, between similar medical technology products, and that this difference can be reliably assessed with relatively uncomplicated empirical methods.
Three of the tested pumps were quite imprecise in the delivery of small quantities of insulin; this would be a problem if they were used on very small children.
The naïve estimates on pumps’ values were very much lower than their actual retail prices; the underestimation must be seen in relation to respondents’ experience as consumers of modern communication technology and the fact that the products tested were paid for by public funds.
A large study of attitudes among 451 young users of CGM systems identified “this was more difficult and complicated than I thought” as the strongest perceived disadvantage after 6 months of use (JDRF CGM Study Group 2008; 2009; 2010). In spite of our knowledge on the importance of usability and its relation to functional use and safety, problems related to the use of medical technology products remain. One reason for this is identified as the inability of manufacturers of medical technology products to understand the advantage of applying formal human factors methodology in the design process (Money et al 2011).
The problem of poorly functioning user interfaces places a heavy responsibility on the purchaser, who must choose the products best fitting the users. There are indications that decisions on purchasing often interpret usability and patient safety as technical reliability and precision (Keselman et al 2003). The design process for medical technology products needs to focus more sharply on the end user, particularly if this is the patient (Berg et al 2003; Nemeth et al 2005).
REFERENCES
Berg, M., Aarts, J., & van der Lei, J. (2003;42(4)). ICT in Health Care: Sociotechnical Approaches. Methods Inf Med, 297-301. Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. Intl. Journal of Human– Computer Interaction, 24(6), 574-594. Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189, 194. FDA (1996) Do it by design: an introduction to human factors in medical devices http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm094957.htm http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm095061.pdf ISO/IEC 25062:2006 Common Industry Format (CIF) for usability test reports. ISO 9241-11, ISO / IEC 9126 part 2 and 3. Jordan, PW (1998) An Introduction to Usability. Taylor & Francis. Juvenile Diabetes Research Foundation Continuous Glucose Monitoring Study Group. (2008). JDRF randomized clinical trial to
assess the efficacy of real-time continuous glucose monitoring in the management of type 1 diabetes: research design and methods. Diabetes Technol Ther, 10; 310-321. Juvenile Diabetes Research Foundation Continuous Glucose Monitoring Study Group. (2009). The effect of continuous glucose monitoring in well-controlled type 1 diabetes. Diabetes Care, 32; 1378-1383. Juvenile Diabetes Research Foundation Continuous Glucose Monitoring Study Group. (2010). Effectiveness of continuous
glucose monitoring in a clinical care environment: evidence from the Juvenile Diabetes Research Foundation Continuous Glucose Monitoring (JDRF-CGM. Diabetes Care , 33, pp. 17-22. Keselman, A., Patel, V. L., Johnson, T. R., & Zhang, J. (2003). Institutional decision-making to select patient care devices: identifying venues to promote patient safety. Journal of biomedical informatics, 36(1), 31-44. Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. In Human Centered Design (pp. 94-103).
Springer Berlin Heidelberg. Money et al (2011) The role of the user within the medical device design and development process: medical device manufacturers’ perspectives. BMC Medical Informatics and Decision Making, 11:15. http://www.biomedcentral.com/1472- 6947/11/15 Nemeth, C., Nunnally, M., O'Connor, M., Klock, P., & Cook, R. (2005). Getting to the point: developing IT for the sharp end of
healthcare. J Biomed Inform , Feb;38(1), 18-25. NHS (2010) Design for patient safety: user testing in the development of medical devices http://www.nrls.npsa.nhs.uk/resources/?EntryId45=74946 Sauro, J. (2011). Measuring usability with the system usability scale (SUS). (access 2012-06-25)[2011-05-08].
Http://www.measuringusability.com/sus.php
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in Safety Management and Human Factors
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2100-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional