Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Approach to Member Checking Support for Characterizing Team Processes with Discrepant Information about Team Roles

College of Computing and Informatics, Drexel University, Philadelphia, PA 19104, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1005036

Abstract

In order to determine whether a process should be improved, it is important to first be able to characterize it. The characterization process is more difficult with team processes involving multiple roles, especially when every role does not participate in and does not have complete knowledge about each task. Ideally analysts should be able to elicit process knowledge from all participating roles synchronously and with sufficient time to cover all related topics. However practical constraints limit such process knowledge elicitation sessions and therefore lead to discrepant information across data collection instances. We present an approach for automating the identification of discrepant data related to role information for the associated team process. The approach includes highlighting the strength of the evidence derived from the interviewees and thereby identifying opportunities for member checking based on conflicts between those who are classified as providing strong evidence for a particular task and based on the need for confirming information when all of the interviewees are classified as providing weak evidence for a particular task.ACKNOWLEDGEMENTSThe National Heart, Lung, and Blood Institute (NHLBI) funded this study through an investigator-initiated research grant (1R01HL153735; PI: Lane-Fall).

INTRODUCTION

In order to determine whether a process involving human performers should be improved, a first important step is to characterize it. Characterizing team processes with many roles of varying responsibilities is difficult. Team pro- cesses include a range of possibilities such as human performers addressing different goals across time, independent and dependent individual and group activities that are distributed across physical locations and phases of the pro- cess, different sets of roles that participate in and across different phases, and varying information requirements. The influence of contextual factors such as priority, availability of resources, knowledge and experience of team members, and environmental features can add contingencies to an otherwise nominal process. This situation leads to the need to collect data from many sources.

© 2024. Published by AHFE Open Access. All rights reserved. 46

Approach to Member Checking Support for Characterizing Team Processes 47

When characterizing a team process, analysts want to ensure that the col- lected information about each task comes from valid sources who know the critical details. For example, information and insights from subject matter experts (SMEs) about tasks in which they currently participate may tend to be more valid than from those who watch a task being completed by others or who have not been responsible to execute the task recently.

Due to the need for valid information and as individual team members’ knowledge of every task in the complete process is likely limited, process analysts solicit information from many role representatives (where a repre- sentative works in a particular position or job role with associated duties, authorities, and accountabilities). Ideally analysts should be able to elicit pro- cess knowledge from all participating roles synchronously and with sufficient time to cover all related topics. In this way, the analysts can identify accurate and complete information about all tasks. Any discrepancies between SMEs in reporting task and related information can be discussed with all participat- ing roles and thus resolved immediately. Unfortunately, staffing, scheduling, and other constraints often limit such knowledge elicitation sessions such that characterization of team processes generally is completed in multiple sessions, followed by rounds of member checking and related activities (Creswell and Poth, 2017) to address missing and conflicting information.

Analysts would benefit from support to organize and to integrate collected team process information from multiple knowledge elicitation sessions with different sets of role representatives. Such support would be especially help- ful when SMEs provide conflicting information or when individual SMEs provide incomplete information. For example, automating the identification of missing information from roles that participate in tasks would be of great benefit. In addition, helping the analysts to identify conflicting information between different role representatives would be helpful.

As part of a larger study focusing on implementing an evidence-based oper- ating room to intensive care unit handoff process (Lane-Fall et al., 2021), researchers developed knowledge elicitation guides, visual and Microsoft® Excel spreadsheet representations of team processes, and qualitative data analysis methods to facilitate interviewing and analyzing team process data from multiple roles (Hose et al., 2022). For each team process under con- sideration, researchers interviewed SMEs individually and in groups with multiple roles. After cleaning interview transcripts, researchers annotated them with role and task related information. To complete the analysis tasks, the researchers used structured Microsoft® Excel spreadsheets to capture task and role specific information and also represented the team process as a process map using Lucidchart. However no tools could be found to help analysts organize the team process data across interviews, to identify dis- crepancies when SMEs from similar and different roles disagreed about role and/or task data, and to identify when no SME who participated in the task provided confirming evidence. With respect to discrepancies, no tool high- lights discrepancies based on whether or not the disagreeing SMEs participate in the task (thereby providing stronger evidence).

This manuscript describes concepts to support automating the identifi- cation of discrepant data related to role information for associated team

processes, and for highlighting the strength of the evidence derived from the SMEs. The idea is to identify opportunities for member checking based on conflicts between SMEs who are classified as providing strong evidence for a particular task and based on the need for confirming information when all of the interviewed SMEs are classified as providing weak evidence for a particular task.

COLLECTING, ANNOTATING, AND REVIEWING TEAM PROCESS DATA Analysts solicit team process data in order to characterize the process. In this work, the SMEs describe the team process with respect to the following (Hose et al., 2022):

  • Set of tasks: description of each task,
  • Task to phase: what task(s) occur in each phase,
  • Task to role: what role(s) are responsible to execute each task,
  • Task to information: what information is acquired, processed, and recorded in each task, and
  • Task to technology and tools: what artefacts support each task.

To facilitate data collection, one strategy is for SMEs to view a reference process map with an initial set of tasks with descriptions that include the relevant information addressed, task to phase mapping, task to role mapping, and identifications of artefacts that support task execution (see Figure 4 in Lane-Fall et al., 2021 for an example). The analysts can then verify the data in the process map with the SMEs. For each SME who provides data, analysts note what tasks have no differences with the reference process map, what tasks are completed in the reference but not at the SMEs’ organization, and what changes there are for the task descriptions and mappings.

With input of structured data describing the reference process map, our custom set of Python (Van Rossum et al., 2023) scripts extract and validate the analysts’ notes and create a new Microsoft® Excel spreadsheet with the new phase, task, role, information, and artefact data. An analyst can use the data to draw an updated process map and for member checking.

Identifying the need for member checking requires tediously reviewing the spreadsheet data entries. In particular, when there are disagreements between SMEs of different roles who all participate in the task, the analysts should engage in member checking with representatives of those multiple roles, a process potentially difficult to schedule as a synchronous meeting. Supporting such analysis tasks inspired us to develop an approach and associated tools.

HIGHLIGHTING DISCREPANCIES AND THE STRENGTH OF THE EVIDENCE TO INFORM MEMBER CHECKING There are multiple reasons why an analyst may wish to conduct member checking after collecting data about a task in a team process. One reason is when none of the interviewed SMEs participate in the described task. In this

Approach to Member Checking Support for Characterizing Team Processes 49

case, gaining confirmation from a role who participates would be advanta- geous. Another case is when at least two interviewed SMEs who participate in the task disagree with each other. Further discussions could identify addi- tional context or other information to shed light on why the SMEs disagree. Another case includes the situation where at least one SME who participates in the task disagrees with at least one SME whose role does not participate in the task. Here one could privilege the information from the participating SME as this SME is likely more familiar with the details of the task and its execution.

Gleaning these types of situations in qualitative data sets can be compli- cated, especially when the data are collected in multiple knowledge elicitation sessions. Our team developed logic and are developing a toolset to support such analyses.

Consider an example focusing on the elicitation of role information, sketched in Figure 1. Here we have four roles who provide role participation information about a single task. A trainer states that the “Trainer” role par- ticipates with a “Designer.” Thus as the “Trainer” role participates, this role is classified as strong with respect to providing information about the task. A logistics representative provides similar role information as the trainer. The logistics representative and the trainer are therefore in the same role agree- ment group. However the logistics representative is classified as “Weak”with respect to participation as this role does not participate in the task. A designer states that the “Designer” role participates with a “Scribe.” This situation creates a separate agreement group from the trainer and the logistics repre- sentation. The “Designer”role is classified as strong with respect to providing information about the task (due to stating participation in the task). The scribe provides different role information than the other two role agreement groups and is assigned to a third agreement group.

Figure 1: Hypothetical example with four roles (trainer, designer, logistics represen- tative, and scribe). Two role representatives state participation in the task and are classified as “strong” (trainer and designer). The role representatives collectively state that three different role sets participate and thus there are three agreement groups. Two of these agreements groups are “strong” because each of the two strong role representatives belong in a different role agreement group. This situation creates a strength of evidence conflict as two strong role agreement groups disagree.
Original PDF Figure 1:

In this example, there are two strong agreement groups, and therefore there is a what we call a strength of the evidence conflict. Here two SMEs who par- ticipate in the task state that different role sets participate. This is a situation where member checking is prudent.

Figure 2 summarizes the more general logic for assessing the strength of the provided evidence and the need for member checking. Each interviewed SME providing information about a task is assigned to a Task Information Providing Role (TIPR). As highlighted above, if a SME states participation in the task, the TIPR is classified as “Strong”. Considering only the role infor- mation, the TIPRs are assigned to “Role Participation Agreement Groups (RPAGs)” based on providing the exact same role set (such as agreement groups 1, 2, and 3 in Figure 1). The RPAG is classified as “Strong” if at least one TIPR is “Strong”.

Figure 2: Strength of evidence and agreement logic based on whether the subject matter experts agree and whether each makes a statement about participating in the task.
Original PDF Figure 2:

There is one RPAG if everyone agrees (bottom right of Figure 2). Here there are no conflicts between the interviewed SMEs. However, if none of the interviewed TIPRs are “Strong”, then it would be useful to get confirming information from a participating role when member checking.

If there are multiple RPAGs and if all of the TIPRs are weak, then there is a need to get confirming information from a participating role when member checking (upper right of Figure 2). The number of RPAGs determines if there is a single conflict or multiple where the latter may take more effort during member checking.

If there are multiple RPAGs and only one is “Strong”, then there is a con- flict but one could decide to privilege the information from the latter. That is, the analyst may choose to consider information coming from the SME who participates in the task as strong evidence. If more than one RPAG is

Approach to Member Checking Support for Characterizing Team Processes 51

strong, then there are two or more strength of the evidence conflicts. Here the member checking process is important because at least two SMEs from participating roles are stating different facts. Getting resolution on this type of disagreement may be a higher priority than the others, given the sources of the information.

CONCLUSION

When characterizing a team process, analysts may collect data from multi- ple sources, some of whom may provide incomplete information and others of whom may disagree with each other about task details. We present an approach for automating the identification of discrepant data related to role information for associated team processes, and for highlighting the strength of the evidence derived from the SMEs. The approach identifies opportuni- ties for member checking based on conflicts between SMEs who are classified as providing strong evidence for a particular task and based on the need for confirming information when all of the interviewed SMEs are classified as providing weak evidence for a particular task. We have a similar approach for other elements of data such as the description of the task.

As of this writing, we are in the code implementation phase for the member checking support. We plan to complete the implementation phase and to test the tool set with data from our operating room to intensive care unit study. We also plan to integrate the member checking support with our transcript analysis tool set. The end goal of this work is to contribute tools that can support analysts completing qualitative data analysis to inform team process improvement.

ACKNOWLEDGMENT

The National Heart, Lung, and Blood Institute (NHLBI) funded this study through an investigator-initiated research grant (1R01HL153735; PI: Lane- Fall). The content is solely the responsibility of the authors and does not represent the official views of the NHLBI. The authors would like to acknowl- edge Bat-Zion Hose, Julian Conn Busch, Eric Savoy, and Alexis Upshur for contributing to the development of the manual process for analyzing tran- scripts from team process knowledge elicitation sessions. The authors would like to acknowledge Anthony Chen, Kenny Huang, Ahn “Oliver” Nguyen, Leighland Rodrigo, and Nico Saputra for contributing to the development of earlier prototypes of the team process characterization software.

REFERENCES

Creswell, J. W. & Poth, C. N. (2017). Qualitative inquiry and research design: Choosing among five approaches. 4th ed). Thousand Oaks, CA: Sage Publications. Hose, B-Z, Busch, J. C, Lane-Fall, M. L., & Bass, E. J. (2022). Systematic approach for

comparing team-based care processes. The 66th International Annual Meeting of the Human Factors and Ergonomics Society, Atlanta, GA, October 10-14, 2022, 580–584.

Lane-Fall, M. B., Christakos, A., Russell, G. C., Hose, B-Z, Dauer, E. D., Greilich, P.

E., Mershon, B. H., Potestio, C. P., Pukenas, E. W., Kimberly, J. R., Stephens- Shields, A. J., Trotta, R. L., Beidas, R. S. & Bass, E. J. (2021). Handoffs and transitions in critical care—understanding scalability: Study protocol for a multicenter stepped wedge type 2 hybrid effectiveness-implementation trial. Implementation Science, 16(1), 63. Van Rossum, G. & Python development team (2023). The python language reference.

Release 3.9.16. Python Software Foundation. Available at https://scicomp.ethz.c h/public/manual/Python/3.9.9/reference.pdf

Cover of the proceedings volume Human Factors in Software and Systems Engineering

Explore the complete digital volume

Human Factors in Software and Systems Engineering

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-16-8

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →