Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Exploring Macrocognitive Healthcare Work: Discovering Seeds for Design Guidelines for Clinical Decision Support

1 Perigean Technologies LLC Fredericksburg, VA 22553, USA ↗
2 Institute for Human and Machine Cognition, Pensacola, FL 32502, USA ↗
3 Cognitive Medical Systems Inc., San Diego, CA 92121, USA ↗
4 Center for Research and Innovation in Systems Safety, Vanderbilt University, Nashville, TN 37232, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe100532

Abstract

Modern healthcare can be described as a macrocognitive work system. In such systems, people seek to adapt to complexity through functions such as sensemaking, coordinating, and re-planning. Technologies can augment human cognitive abilities for managing attention, recalling information, projecting trajectories, and achieving common ground. The ideal healthcare system would integrate technologies in ways that maximize the ability of humans to adapt to complexity through coordinating and synchronizing activities, and help people anticipate surprise and error – while preserving the clinical experience among clinicians and patients. This paper reports on our effort to explore the macrocognitive work undertaken at a major healthcare network in the United States. Our team conducted 60 cognitive interviews across seven facilities, covering inpatient, outpatient and community-based settings. We report on our approach based in methods of Cognitive Task Analysis. The approach was simultaneously structured and adaptable, and was therefore well suited for exploratory data collection, as it permitted adjustments to the data collection strategy across a wide spectrum of performers, experience-levels, and work contexts. We review our data collection, analysis and representation methodology, and the seeds for design guidelines for clinical decision support that resulted from the effort.

INTRODUCTION

Modern healthcare can be described as a macrocognitive work system. In such systems people seek to adapt to complexity through functions such as sensemaking, coordinating, and re-planning (Hoffman et al., 2009). Macrocognitive work may be undertaken individually (e.g., forming mental models about patients, detecting problems), organizationally (coordinating care, maintaining common ground), or at both levels (re-planning, monitoring). Macrocognitive work may be informed and facilitated (but in many cases is hobbled) by technological infrastructure: sensors, data repositories, alarms, and communication tools. Technologies can augment human cognitive abilities for managing attention, recalling information, projecting trajectories, and raising awareness. The ideal macrocognitive healthcare system would integrate technologies in ways that maximize the ability of people to adapt to complexity through coordinating and synchronizing activities, and help people anticipate surprise and error (Woods, 2005) — all while preserving and enhancing the clinical experience between clinicians and patients. In other words, provide true clinical decision support.

This paper reports on our effort to explore the macrocognitive work undertaken at a major healthcare network in the United States, in order to jumpstart the delineation of guidelines for the design of clinical decision support. Our effort was part of a larger program to introduce the User-Centered Design process (Weinger, 2013; shown in Figure 1) into the development of clinical decision support capabilities that may be used in the healthcare network we explored, and beyond.

We first describe our interviewees. We then describe our method, data analysis and representation approaches. We close with an overview of the design seeds that are serving as a resource for the development of guidelines for the design of clinical decision support systems.

Figure 1: User-Centered Design Process
Original PDF Figure 1:

PARTICIPANTS AND INTERVIEWS The participants worked in a variety of patient care environments in including the Emergency Department, In- Patient Wards (e.g., Mental Health), Out Patient Primary Care Clinics, and Specialty Clinics. Mr. Moon and Dr. Hoffman, with support from Mrs. Mary Lacroix, conducted 60 interviews across seven sites, as shown in Table 1.

Table 1: Number of Participants by Site, Site Type, and Average Experience Level Site Type Average Experience Number of Participant
Original PDF Table 1:

The qualifications of the interviewees also represented a wide spectrum, as shown in Table 2.

Table 2: Number of Participants by Site, Site Type, and Experience Level Qualifications
Original PDF Table 2:

The duration of the interviews differed, from a minimum of 22 minutes to a maximum of 90. Total interview time amounted to nearly 3,000 minutes (2,949), and the average duration of the interviews was 49 minutes.

METHOD

Cognitive Interviews Mr. Moon and Dr. Hoffman conducted interviews using structured cognitive interviewing based on their experience at Cognitive Task Analysis (CTA; Crandall, Klein, & Hoffman, 2006). Structured cognitive interviewing uses knowledge elicitation probe questions that are designed to elicit a descriptive account of the cognitive functions and processes required to make decisions and perform complex tasks. The cognitive activities elicited include subtle perceptual judgments, assessment of complex and dynamic situations, decision-making, problem solving, anomaly detection, and planning. Cognitive interviewing uncovers the cues, expectancies, goals, strategies, and typical actions taken by domain practitioners. Importantly, cognitive interviewing focuses on aspects of cognitive work that are influenced and enabled by technologies (e.g. health care software and medical information systems)

We used cognitive interviewing to elicit descriptions of the cognitive and collaborative work of healthcare professionals, the technologies and tools that they use, and difficult cases they have encountered. Our cognitive interviewing approach was both structured and adaptable, and was therefore well suited for exploratory data collection. Given the exploratory goals of our effort, we recognized the need to adjust the interviewing approach and methodology across the spectrum of interviewees, experience-levels, and work contexts. The cognitive interview approach enabled the participants to articulate descriptions of these dimensions, and prompted them to cite examples. Examples of the dimensions of macrocognitive work that we explored are provided in Table 3, along with examples of the questions that we asked.

Table 3: Macrocognitive dimensions of performance, and Examples of Generic Questions Macrocognitive Generic Questions
Original PDF Table 3:

Where were the bottlenecks on the team? In addition to these probing questions, we also drew on “cardinal issues of decision making” for probes based in the literature on judgment and decision-making. The analysis of decision making from the sociotechnical perspective (see Hoffman and Militello, 2008) regards individual decisions as nested in layers of organizational and system complexity. There are a number of “cardinal issues” that are implicated in all decisions (Hoffman, & Yates, 2005; Yates, 1990). These became very salient in the present research, as the interviewees began to “tell stories” about individual decision situations: Why is anything being decided? What are the roles and responsibilities? (Additional Cardinal Issue probes appear below, in Table 4.) Stemming from our very first interviewing results—showing that individual decisions are best understood in their broader organizational and episodic contexts—we used these cardinal issues to extend the set of probes and to form a set of additional categories that were utilized in the analysis of the interview protocols.

Where appropriate and feasible, we also we executed the Critical Decision Method (CDM), which is adapted from the literature on Naturalistic Decision Making (Crandall, Klein, & Hoffman, 2006, p. 69-90). CDM is a structured interviewing protocol for eliciting and organizing stories. Critical Decision Method has been effectively used to elicit stories about specific medical practices, for example emergency physician expertise (Schuber et al., 2013). We found it interesting that the standard initial CDM probe, "Can you tell me of a recent case that was especially difficult in terms of your decision making?”, often did not result in a response that suggested that the CDM was appropriate. This finding was, in itself, interesting, as it suggested that in most cases, clinical decision-making was not difficult. Rather, other aspects of their overall work system, to include technologies and tools, were difficult to deal with.

Observations Contextual Inquiry Observations (CIO) were originally proposed as a data collection technique for this effort. However, cognitive interviews were deemed a more appropriate fit in order to mitigate confidentiality concerns and to enable flexibility in scheduling. Moreover, cognitive interviews took account of the workplaces, artifacts, and strategies that would have been captured using CIO, but also allowed for deep dives into areas of interest. That said, we collected some observations of in situ workplace cognition where it was directly relevant to issues being explored in the cognitive interviews. For example, while interviewing a nurse regarding her role in patient transfers, we observed about five interruptions from colleagues seeking her support – observations that provided invaluable insight into the challenges of information management, collaboration, and interruption for her role.

Data Analysis and Representation Coding. From the 60 interviews, Mr. Moon and Dr. Hoffman selected a subset for detailed data analysis. All interviews were reviewed and rated with regard to their relative “value” to our purpose. Interviews were rated “1” if their content reflected high value insights by virtue of a unique perspective of the interviewee (e.g., director position, consult provider, new participant role), extended duration, and the depth of perspective of the interviewee into their own and their colleagues’ macrocognitive work patterns. Interviews were rated a “2” if their content reflected a previously-explored perspective, abbreviated duration, and/or contained findings that were additive to previously acquired findings. Interviews for which Value=1 were analyzed using two widely-used and complimentary data analysis techniques: Decision Requirements Tables (DRTs) and Concept Maps (Cmaps). Both methods enable data organization, and provide a descriptive analysis of the interview data. These are described below.

In addition to the DRTs and Cmaps, we iteratively developed a macrocognitive work coding scheme. Starting from a notional scheme based in prior research, the Cardinal Issues of Decision Making, and our knowledge of Cognitive Systems Engineering (Hoffman, 2012), we generated, then applied and iteratively refined a coding scheme that focused on issues in decisions about patients, and issues in the macrocognitive work. Like all coding schemes, ours was more or less useful, depending on the data collected in any given interview. We offer the scheme in Table 4 for use by other researchers.

Table 4: Final Coding Scheme
Original PDF Table 4:

Decision Requirements Tables. The DRT is a data summarization analysis approach that organizes findings into descriptions of the key and supporting decisions, and how they are made (Crandall, Klein, & Hoffman, 2006). The DRT organizes data around the decisions that describe the decision context, considerations and approaches taken by performers. Specifically, the fields of data organization used to analyze interviews are shown in Table 5.

Table 5: Decision Requirements Table Categories Decision Requirements Definitions
Original PDF Table 5:

Concept Maps. Concept Maps are an original type of meaningful diagram that are used to represent and convey knowledge (Moon, Hoffman, Novak & Canas, 2011). Concept Maps involve labeled nodes and links. The nodes represent concepts, which are enclosed in boxes. The label for most concepts is a word or just a few words, although one can also use symbols. Concepts are related to one another by meaningfully labeled linking lines (Crandall, Klein, & Hoffman, 2006, p. 51). While Cmapping can be used as a knowledge elicitation method, Cmaps are also useful for representing descriptive data derived by using other CTA methods. They can show, at a glance, the domain knowledge, processes, and illustrative quotations. Most importantly, they serve to organize findings in a semi-hierarchical structure, enabling appreciation of the “gist” of the findings. Whereas DRTs provide a highly structured, detailed analysis of decisions in a tabular format, Cmaps provide a graphical depiction of decisions and their contextual constraints and affordances. Additionally, Cmaps can provide a hyperlinked organization of the data, allowing for the demonstration of connectivity between subsets of findings. A full appreciation for the interconnectedness of Cmaps is only possible by viewing the Cmaps in hyperlinked, digital form.

Concept Maps have been used in qualitative healthcare research to, for example, describe expert decision making of anesthesiologists (Weinger, 2013), understand comprehension of patient education (Marchand et al., 2002), and accelerate knowledge acquisition regarding complex medical devices (Barberá-Thomás et al., 2011).

We created seven sets of Cmaps covering 7 interviews (not analyzed using DRTs), totaling 37 Cmaps. The Cmaps included descriptions of timelines and decision points, challenges and design opportunities. Three examples are shown below, in Figures 2 through 4.

Figure 2. Hybrid Concept Map and Decision Timeline
Original PDF Figure 2.
Figure 3. Concept Map describing challenges and design opportunities for Tumor Boards
Original PDF Figure 3.
Figure 4. Concept Map describing challenges in narcotics prescribing MAIN FINDINGS
Original PDF Figure 4.

Contextual Understanding Macrocognitive Workflows. The DRTs and Cmaps described above provide analyzed and organized descriptions of the interview data, and thereby describe the macrocognitive work explored with individual interviewees.

A second set of Cmaps was also generated to provide a set of “workflows” describing macrocognitive work across interviewees. It is imperative to describe what is meant by “workflows” in the context of our effort. Our use is akin to Gruman et al.’s (2013) concept of “mindflows”— the patterns of thinking that knowledge workers use while doing their work. Our workflows describe a set of macrocognitive areas of performance that comprise work in the healthcare domain, regardless of role. They describe what knowledge is required to conduct the work, the contextual barriers that make the cognitive work difficult, approaches for achieving the cognitive work and managing the cognitive workload, and potential (negative) outcomes of the cognitive work. They provide context for understanding how work is accomplished, and thus can be useful for informing design.

The workflows were not intended as process descriptions for the management of people, information, or processes. Nor are they an indictment on any particular healthcare system, policy, or practice. Rather, they describe—in part through the words of our participants—challenges to cognitive work that are difficult by their nature, but that are sometimes made especially difficult because of organizational requirements and constraints, or that are often made more difficult because of usability and usefulness gaps in the information systems. In many cases, macrocognitive work is challenging because of all three.

Our set of Macrocognitive Workflows is described in Table 6. Two examples are shown in Figures 4 and 5.

Table 6: Macrocognitive Worflows
Original PDF Table 6:
Figure 4. Macrocognitive Workflow - Assessing
Original PDF Figure 4.
Figure 5. Macrocognitive Workflow - Diagnosing
Original PDF Figure 5.

Stories. The challenges of implementing CDM were noted above. We were able to execute a complete CDM protocol in 10 of the interviews. Thus, the dataset includes 10 detailed stories of clinical decision making across several of the workflows. The stories are too lengthy to show here. Example topics are shown in Table 7.

Table 7: Critical Decision Method Stories Macrocognitive Story
Original PDF Table 7:

Desirements, Workarounds, Kludges, and Make-work. The cognitive interviews enabled close inspection of macrocognitive work, including work that involves reliance on computer systems. Naturally, the interviews touched on Participants’ desirements for new technologies. Desirements are functionalities that support the macrocognitive work, have not been incorporated in a system build or deployment, and represent important operational capabilities. Hence, desirements are not the same as “requirements” (see Hoffman and McCloskey, 2013). In addition to learning what capabilities and functionalities providers would want in their technologies, we also learned a great deal about the workarounds, kludges and make-work they had developed in order to achieve their goals using the systems that they had at their disposal. Whereas workarounds are procedural deviations that must be implemented to circumvent or compensate for a design flaw, a kludge is a system made of components that are poorly matched or were originally intended for some other use (Koopman and Hoffman, 2003). Make-work is repetitive, boring, time- consuming activities that someone must engage in to accomplish something that could not be accomplished using a shortcut, or that one should be able to easily accomplish but cannot. Our analysis and representation products are replete with examples of commentary about these kinds of design issues.

Decision Support Seeds Patterson et al. (2001) conceptualized design seeds as a modular strategy for aiding performers around a domain- specific leverage point that is expected to usefully support performance—in contrast to traditional systems engineering approaches. Design seeds can be derived from an exploration of macrocognitive work, individually evaluated for their usefulness, and iterated through the User-Centered Design process.

From our exploration, we derived a set of decision support seeds. The seeds were primarily generated from interview data where interviewees expressed desirements for new or evolved support features, and workarounds, kludges and make-work. Seeds were also abstracted from the decision requirements, in particular the challenges, information requirements, strategies and common errors expressed in the DRTs, Concept Maps, and Macrocognitive Workflows. We organized the seeds mostly by the Macrocognitive Workflows, shown in Table 8, below.

We do not believe our seeds comprise a comprehensive set. While they are as broad as the range of experience of our participants, they are only as deep as the experiences we discussed in our limited time with them. Since our aims were broad and exploratory, we did not deep dive into any particular experience bases. The seeds are also limited by our ability to abstract design ideas from concrete experience—additional sweeps through the interviews would likely yield additional seeds. Moreover, our design seeds are not intended to provide specific guidance for any particular type of system, technology or application. Rather, they served the purpose of advancing our exploratory goal, while suggesting directions to designers focusing on designing decision support the Macrocognitive Workflows.

In particular, we believe that several of the findings offer are interesting, generally novel findings that have not been discussed in the literature. These are highlighted in bold in Table 8. We suggest these seeds may offer innovative directions for design that otherwise may not be considered. Notably, these seeds offer support for coordinating and synchronizing activities, and helping people anticipate surprise and error. Uncovering these sorts of seeds strengthens the validity of our structured yet adaptable approach; iterative design and favorable evaluation of them would bolster the utility of the User-Centered Design approach.

Table 8: Decision Support Seeds Workflow Sub-Issue Decision support concept
Original PDF Table 8:

Miller has prepared draft guidelines, which are serving as the basis for designing, evaluating, and iterating a set of CDS interventions (Miller, 2014). Putting the User Centered Design Cycle through its paces in this manner will enable further refinement of the guidelines. Ultimately, the guidelines will provide practical guidance enhancing the User Centered Design, particularly during the design and evaluation phases.

CONCLUSIONS

Modern healthcare must balance three things: (1)the focus on safety and reliability (2) the introduction of software- based systems and (3) the need to mitigate disruptions to the clinical experience. To achieve this balance, the design of healthcare information systems and clinical decision support systems must include an appreciation of macrocognitive work in context. It is only through such an understanding the designers can hope to create truly supportive clinical decision aids, and provide access to the right information, at the right time, for the right people. Cognitive interviewing is necessary as a window onto practitioners’ macrocognition, their desirements, and the decision challenges they face. Decision Requirements Tables and Concept Maps are useful tools for analyzing and organizing qualitative data that informs design and helps to establish guidelines for effective designs. We encourage the extension of our approaches, design seeds, and further analysis of the guidelines this project helped to shape.

REFERENCES

Barberá-Tomás, D., Schachter, M. E. E., & de los Reyes-López, E. (2011). Improving Organizational Learning with Concept Maps: A Business Case Study. Applied Concept Mapping: Capturing, Analyzing, and Organizing Knowledge, 253. Crandall, B., Hoffman, R., & Klein, G. (2006). Working Minds: A Practitioner's Guide to Cognitive Task Analysis.

MIT Press. Cambridge, MA. Gruman, G., Baya., V., and Parker, B. (2013). The future of enterprise apps: Moving beyond workflows to mindflows. PwC Technology Forecast 2013. Issue 3, pp. 6-19. Hoffman, R. (2012). Collected Essays on Human-Centered Computing, 2001-2011. New York: IEEE Computer

Society Press. Hoffman, R., and McCloskey, M. (2013). Envisioning Desirements. Hoffman, R. R., & Militello, L. G. (2012). Perspectives on cognitive task analysis: Historical origins and modern communities of practice. Psychology Press. Hoffman, R., Norman, D., & Vagners, J. (2009). “Complex Sociotechnical Joint Cognitive Work Systems"?.

Intelligent Systems, IEEE, 24(3), 82-c3. Hoffman, R. and Yates, J. (2005). Decision(?)Making(?). IEEE: Intelligent Systems, pp. 22-29. Koopman, P., & Hoffman, R. R. (2003). Work-arounds, make-work, and kludges. Intelligent Systems, IEEE, 18(6), 70-75. Marchand, C., d'Ivernois, J. F., Assal, J. P., Slama, G., & Hivon, R. (2002). An analysis, using concept mapping, of

diabetic patients' knowledge, before and after patient education. Medical Teacher, 24(1), 90-99. Miller, A. (In Press). Guidelines for Clinical Decision Support. Vanderbilt University. Moon, B., Hoffman, R., Novak, J. & Cañas, A. (2011). Applied Concept Mapping: Capturing, Analyzing, and Organizing Knowledge. New York: CRC Press. Neville, K., Hoffman, R. R., Linde, C., Elm, W. C. and Fowlkes, J. (2008, January/February). The procurement

woes revisited. IEEE Intelligent Systems, pp. 72-75. Osheroff, J., Teich, J., Levick, F., & Robert, A. (2012). Improving outcomes with clinical decision support: an implementer's guide. Patterson, E. S., Woods, D. D., Tinapple, D., & Roth, E. M. (2001, October). Using cognitive task analysis (CTA) to

seed design concepts for intelligence analysts under data overload. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 45, No. 4, pp. 439-443). SAGE Publications. Schubert, C. C., Denmark, T. K., Crandall, B., Grome, A., & Pappas, J. (2013). Characterizing novice-expert differences in macrocognition: an exploratory study of cognitive work in the emergency department. Annals of emergency medicine, 61(1), 96-109. Weinger, M. (2013). Perils and Pitfalls of Anesthesia Displays. Paper presented at the Annual Meeting of the

Society for Technology in Anesthesiology. Woods, D. (2005). Generic support requirements for cognitive work: laws that govern cognitive work in action. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 49, No. 3, pp. 317-321). SAGE Publications. Human Aspects of Healthcare (2021)

Yates, J.F. (1990). Judgment and Decision Making. Englewood Cliffs, NJ: Prentice Hall.

Cover of the proceedings volume Advances in Human Aspects of Healthcare

Explore the complete digital volume

Advances in Human Aspects of Healthcare

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-4951-2093-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →