Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Evaluating the Efficacy of Structured Analytic Techniques (SATs) as a Support System to Enhance Decision-Making within ISR Mission Environments

1 711 th Human Performance Wing, Airman Systems Directorate, WPAFB OH 45433, USA ↗
2 Parallax Advanced Research, Beavercreek, OH 45431, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1004458

Abstract

U.S. Air Force military operators involved with Intelligence, Surveillance, and Reconnaissance (ISR) missions are required to process, exploit, and disseminate (PED) collected intelligence within friendly and hostile environments in near-real time in order to provide geographical locations and ground movement patterns. Intelligence collected during ISR operations are then implemented into future strategic planning to provide our military an edge in the battlefield. However, the information collected can be vague, incomplete, or ill-defined resulting in operators making poor or inadequate decisions. Therefore, the objective of this study was to evaluate the effectiveness of two structured analytic techniques (SATs) against a control group when interpreting and comprehending narrative content in order to support and facilitate current tool development and future technology transition within the ISR community. Three groups of 25 participants (N=75) were randomly assigned to one of two analytic techniques or a control approach and provided a narrative. The SATs implemented were the Method for Defining Analytical Questions (MDAQ) which was developed in-house by our ISR subject matter experts (SMEs), a Scaffolding approach, or a Control approach. MDAQ is a repeatable process focused on identifying an indicator and its association to a person, place, or event before providing a solution. Scaffolding is founded on determining a problem statement, generating a solution, providing justification, evaluating the hypothesis, and providing a solution. For the Control approach, participants read through the content and provided a solution. The objective of the study was to determine if providing a structured analytic technique would enhance the detection of essential elements of information (EEI) embedded within the narrative leading to improved performance accuracy. The findings provided underlying evidence that implementing a Scaffolding approach significantly improved performance accuracy compared to MDAQ and Control (p<0.01). Moreover, a statistically significant difference was detected within the MDAQ group when participants repeated the process compared to those who only went through the process once (p<0.01). Nevertheless, the findings suggests that providing participants with a structured analytic technique enabled them to identify and interpret critical EEIs that maybe overlooked otherwise resulting in improved performance accuracy. This discovery will support human-computer interactions for future ISR tool development.

© 2023. Published by AHFE Open Access. All rights reserved. 238

Evaluating the Efficacy of Structured Analytic Techniques (SATs) 239

INTRODUCTION

Intel analysts and strategic planners are rapidly becoming overwhelmed with the substantial quantify of incoming information as Intelligence, Surveillance, and Reconnaissance (ISR) requirements become more prevalent within mil- itary operations. With the advancements in technology, ISR electro-optical infrared (EO/IR) sensors currently have the capability to capture several billion bits of information per section (Martinez, 2010). The increase in intelligence collected places an overpowering demand on the operators to decrypt, analyze, validate, and disseminate in near-real time in order to main- tain space, air, land, maritime, and cyber superiority (see Figure 1). If intel analysts inaccurately process and disseminate their findings from the col- lected data, geographical locations and ground movement patterns of our adversaries’ may not reflect real-world conditions and could be catastrophic for future path planning directives. In addition to this challenge, data col- lected from ISR operations often contain incomplete, imprecise, and vague information which can significantly hinder the decision-making process and lead to cognitive biases. Previous research has discovered that in a complex, low-validity environment, operators deviate from a heuristic approach and resort to pre-existing experiences and knowledge to produce a quick response (Kahneman, 2011). This results in the operator answering a question that was not initially asked or a sub-section of the original question. Therefore, it is essential to determine a pathway forward to address these obstacles and enhance operators’ decision-making process.

Figure 1: Joint all-domain command and control (JADC2) communications involv- ing space, air, land, maritime, and cyber (crsreports.congress.gov/product/ pdf/r/r46725/8).
Original PDF Figure 1:

Structured analytics has been an area of interest within academia and mili- tary over the past several decades to support important decisions in complex and dynamic environments. The premise of following a structured approach is to mitigate the two canonical sources of error which are systematic biases and random noise (Chang et al., 2017). Although, with hundreds of struc- tured analytic techniques (SATs), how do analysts know which technique would optimize processing and comprehension when confronted with ill- defined collected intelligence (Pherson and Heuer, 2020). Previous literature has discovered that using the Analysis of Competing Hypotheses (ACH) approach could support intel analysts reduce pre-existing biases and lead to

improved performance (Jones, 2018). The ACH workflow structure is based upon creating multiple hypotheses, rating the hypotheses on consistencies and inconsistencies, and selecting the most credible option. The issue with using the ACH approach is the inevitability of the workflow structure becoming increasingly large when relating hypotheses and their consistencies and incon- sistencies. This can significantly increase the time required to analyze the intelligence, which typically is not an option within military environments. Another technique, commonly used across a multitude of military platforms to enhance computational thinking and self-efficacy, is the scaffolding tech- nique (Yelland and Masters, 2007). The scaffolding workflow structure is based upon representing the problem statement, generating a solution, justi- fying the selection, and testing and evaluating the results (Javed and Elmqvist, 2013). The issue with using the scaffolding approach, particularly in a tacti- cal and operational environment, is the lengthy time required needed to cycle through the process. Nevertheless, the scaffolding approach has been shown to be more effective than the ACH approach in improving resource acquisi- tion (Omohundro, 2014), imagery interpretation (DeWiggins et al., 2010), and accelerating operators training (Vogel-Walcutt et al., 2011) with military environments.

Despite the underlying evidence that the scaffolding approach appears to provide the most relatable characteristics that can transition into ISR mili- tary operations to improve decision-making capabilities, it is important to determine if a hybrid-approach of these techniques would be better suited for comprehending complex collected intelligence. To address this question, a group of ISR subject matter experts (SMEs) who have a combined 50+ years’ within ISR operations, processes, and procedures came together to discuss the ongoing challenges regarding the breakdown of intel problems, questions, and capability gaps. This yielded the development of a new hybrid-approach known as the Methods for Defining Analytical Questions (MDAQ). MDAQ is founded upon the reduction of priority intelligence requirements (PIRs) into EEIs and then into specific supporting indicators. Before the MDAQ approach can be implemented into a classified military environment, it must be evaluated in a simulated unclassified scenario. The ISR SMEs noted that the tedious process of sorting through and identifying useful versus superflu- ous ISR imagery and full-motion video (FMV) has similar characteristics to an unclassified, ambiguous narrative. As a result, a narrative was discovered which required a human operator to read through the content and identify embedded EEIs. If the EEIs are correctly identified, it can provide underly- ing evidence indicating the correct solution or the best future plan of action. Therefore, the purpose of this research study was to determine if the MDAQ approach could enhance problem-solving efficacy in relation to a scaffolding and control approach.

METHODS AND MATERIALS Participants

The study was approved by the Air Force Research Laboratory Institu- tional Review Board (AFRL IRB) on human participants to evaluate the

Evaluating the Efficacy of Structured Analytic Techniques (SATs) 241

efficacy of SATs when provided an ambiguous narrative. Previous litera- ture discussed the influence of contexts when incorporating a scaffolding approach on cognitive decision-making (Belland et al., 2017). The analy- sis conducted by Belland et al., (2017) discovered a pairwise comparison of 0.27 for pooled standard deviation of subjects and a mean difference of 0.23. Using a standard power of 0.80, a sample calculation was per- formed resulting in a sample size of 25. Therefore, based on the esti- mates presented in Belland et al., (2017), the current study included a total of 75 participants (i.e., 25 per group) which were randomly assigned to a SAT (MDAQ, scaffolding, control) and given the same ambiguous narrative.

Participants were recruited through MTurkTM, which is an online platform where surveys and research experiments can be hosted. Participants were required to be at least 18 years old, speak fluent English, located in the U.S., and have achieved at least a 95% accuracy rate on MTurkTM. This distinction represents that the participant has provided reliable data for their work. Par- ticipants were also provided additional screening and demographic forms to identify gender, age, education level, sleep patterns, and gaming experience. These demographics were selected because previous research has shown a direct correlation with respect to performance during cognitive assessments (Hambrick and Engle, 2002).

Task The task was adapted from an online database (5minutemystery.com). This database is comprised of ambiguous narratives allowing users to evaluate the content with the objective of detecting subtle indicators leading to high confident, accurate decisions. The database was identified and vetted by ISR SMEs as an activity that required similar decision-making characteris- tics for intel analysts within complex military operations. The title of the narrative was masked and the names of the characters were modified to deter participants from potentially searching for answers online. In addition, a readability calculation was performed on the narrative which provided a Flesch-Kincaid Grade Level score. This score depicts the minimum educa- tional grade level recommended for the reader to be able to properly interpret and comprehend the material (Cann et al., 2014). When providing con- tent to the general public the Flesch-Kincaid Grade Level score should be below a 7th grade level in order to improve readability and understanding (Walsh and Volsko, 2008). The overall score for the narrative was at a 5th

grade level.

Within the narrative selected, there were four subtle EEIs that could be identified. Each EEI was directly related to a possible solution. EEI 1,2, and 3 provided insight that would eliminate a particular solution. EEI 4 provided incriminating evidence that would lead to the correct solution. The Flesch- Kincaid Grade Level score was performed on each of the EEIs. The score for EEI 1 was 7th grade, EEI 2 was 7th grade, EEI 3 was 10th grade, and EEI 4 was 12th grade. Therefore, the EEI directly related to the correct solution was the most difficult to detect and identify.

Procedures Participants were able to complete the task online by following a link that was hosted on MTurkTM. Each participant was randomly assigned to either MDAQ, scaffolding, or the control approach and provided the same narra- tive (see Figure 2). They were instructed to read the content and answer the questions based on the SAT provided. Each participant was provided 1 hour to complete the task.

Figure 2: Structured Analytic Techniques (SATs) workflow process and a brief description.
Original PDF Figure 2:

RESULTS

As shown in Table 1, there was a statistically significant main effect between SAT and performance accuracy (F2,72=5.14, p<0.01). As a result, an ANOVA was conducted comparing each SAT and performance accuracy. As shown in Table 2, there was a statistically significant difference between scaffolding and control (F1,48=7.36, p<0.01) and scaffolding and MDAQ (F1,48=7.36, p<0.01). The scaffolding approach provided the correct solution for 15 of the 25 participants (60%) whereas MDAQ and control provided the correct solution for 6 of the 25 participants (24%) (see Figure 3).

As shown in Table 3, there was a statistically significant main effect between SAT and time on task (F2,72 = 5.40, p < 0.01). As a result, an ANOVA was conducted comparing each SAT and time on task. As shown in Table 4, there was a statistically significant difference between MDAQ and control (F1,48 = 9.43, p < 0.01). The MDAQ group required almost double the amount of time to complete the task (approx. 21 minutes on average)

Evaluating the Efficacy of Structured Analytic Techniques (SATs) 243

Figure 3: Performance accuracy for each structured analytic technique (SAT).
Original PDF Figure 3:
Table 1. Analysis of variance (ANOVA) depicting the main effect of structured analytic techniques (SATs) and performance accuracy.

Table available in the original PDF paper.

Approach Source df SS MS F p Between-Conditions 2 2.16 1.08 5.14 <0.01 SATs Within-Conditions 72 15.12 0.21 Total 74 17.28

Statistical Significance at alpha level of 0.05.

Table 2. Analysis of variance (ANOVA) comparing structured analytic techniques (SATs) and performance accuracy.

Table available in the original PDF paper.

Approach Source df SS MS F p scaffolding Between-Conditions 1 1.62 1.62 7.36 <0.01 vs Within-Conditions 48 10.56 0.22 control Total 49 12.18 scaffolding Between-Conditions 1 1.62 1.62 7.36 <0.01 vs Within-Conditions 48 10.56 0.22 MDAQ Total 49 12.18 MDAQ Between-Conditions 1 0.00 0.00 0.00 1.00 vs Within-Conditions 48 9.12 0.19 control Total 49 9.12

Statistical Significance at alpha level of 0.05.

compared to control (approx. 11 minutes on average). Scaffolding required approximately 15 minutes on average to complete the task.

As shown in Table 5, there was a statistically significant main effect between SAT and detection of EEIs for EEI 1 (F2,72 = 3.42, p = 0.04) and EEI 4 (F2,72 = 11.88, p < 0.01). As a result, an ANOVA was con- ducted comparing each SAT and detection of EEI 1 and EEI 4. As shown in Table 6, there was a statistically significant difference between scaffolding and MDAQ (F1,48 = 6.19, p = 0.02) for detecting EEI 1 and a statistically sig- nificant difference between scaffolding and MDAQ (F1,48 = 13.48, p<0.01) and scaffolding and control (F1,48 = 13.48, p < 0.01) for detecting EEI 4. The scaffolding approach was able to correctly detect EEI 4 for 11 of the 25

participants (44%) compared to 1 out of 25 (4%) for MDAQ and control (see Figure 4).

There was no statistically significant difference with respect to demograph- ics (i.e., gender, age, education level, sleep patterns, gaming experience) and self-rated confidence for performance accuracy.

Table 3. Analysis of variance (ANOVA) depicting the main effect of structured analytic techniques (SATs) and time on task.

Table available in the original PDF paper.

Approach Source df SS MS F p Between-Conditions 2 4128505 2064252 5.40 <0.01 SATs Within-Conditions 72 27507749 382052 Total 74 31636254

Statistical Significance at alpha level of 0.05.

Table 4. Analysis of variance (ANOVA) comparing structured analytic techniques (SATs) and time on task.

Table available in the original PDF paper.

Approach Source df SS MS F p scaffolding Between-Conditions 1 756204 756204 3.22 0.08 vs Within-Conditions 48 11272278 234839 control Total 49 12028482 scaffolding Between-Conditions 1 1335305 1335305 2.80 0.10 vs Within-Conditions 48 22866240 476380 MDAQ Total 49 24201545 MDAQ Between-Conditions 1 4101248 4101248 9.43 <0.01 vs Within-Conditions 48 20876980 434937 control Total 49 24978228

Statistical Significance at alpha level of 0.05.

Figure 4: Essential elements of information (EEI) correctly identified by each structured analytic technique (SAT).
Original PDF Figure 4:

Evaluating the Efficacy of Structured Analytic Techniques (SATs) 245

Table 5. Analysis of variance (ANOVA) depicting the main effect of structured analytic techniques (SATs) and essential elements of information (EEI).

Table available in the original PDF paper.

EEI Source df SS MS F p Between-Conditions 2 1.04 0.52 3.42 0.04 1 Within-Conditions 72 10.96 0.15 Total 74 12.00 Between-Conditions 2 0.75 0.37 1.93 0.15 2 Within-Conditions 72 13.92 0.19 Total 74 14.67 Between-Conditions 2 0.56 0.28 1.76 0.18 3 Within-Conditions 72 11.44 0.16 Total 74 12.00 Between-Conditions 2 2.67 1.33 11.88 <0.01 4 Within-Conditions 72 8.08 0.11 Total 74 10.75

Statistical Significance at alpha level of 0.05.

Table 6. Analysis of variance (ANOVA) comparing structured analytic techniques (SATs) and essential elements of information (EEI).

Table available in the original PDF paper.

EEI Approach Source df SS MS F p 1 Scaffolding Between-Conditions 1 0.88 0.08 0.74 0.39 vs Within-Conditions 48 5.20 5.28 Control Total 49 5.28 Scaffolding Between-Conditions 1 0.98 0.98 6.19 0.02 vs Within-Conditions 48 7.60 0.16 MDAQ Total 49 8.58 MDAQ Between-Conditions 1 0.50 0.50 2.63 0.11 vs Within-Conditions 48 9.12 0.19 Control Total 49 9.62 4 Scaffolding Between-Conditions 1 2.00 2.00 13.48 <0.01 vs Within-Conditions 48 7.12 0.15 Control Total 49 9.12 Scaffolding Between-Conditions 1 2.00 2.00 13.48 <0.01 vs Within-Conditions 48 7.12 0.15 MDAQ Total 49 9.12 MDAQ Between-Conditions 1 0.00 0.00 0.00 1.00 vs Within-Conditions 48 1.92 0.04 Control Total 49 1.92

Statistical Significance at alpha level of 0.05.

CONCLUSION

Within an ISR environment, the slightest misinterpretation can lead to poor decisions and can influence the success or failure of current mission objec- tives, the safety and well-being of military personnel, and the ability to maintain military superiority. Therefore, it is imperative that we understand

how data is collected and fused, as well as the validity of the findings through human factors interpretation. Many efforts have been made to improve information processing capabilities based on complex and unstructured infor- mation over the years, although few are focused on the ISR community (Nelson, 2016). The purpose of this research study was just that, examining if SATs can enhance critical thinking when confronted with vague information similar to an ISR environment.

The findings provided underlying evidence that a scaffolding approach was able to significantly improve performance accuracy when analyzing an ambiguous narrative, which closely resembles the process of breaking down PIRs into EEIs by intelligence analysts. Our findings coincide with a recent study that discovered implementing a scaffolding approach signif- icantly improved readability and understanding when interpreting textual content by building upon exiting knowledge and enabling critical thinking (Jirasatopron and Hiranburana, 2016). However, it has also been discov- ered through previous research that as the readability and complexity of the content increases, scaffolding may be limited in providing support and enhancing comprehension (Ge and Land, 2003). Future research should be conducted comparing novices and exerts on military and non-military per- sonnel as the Flesch-Kincaid Grade Level score increases before transitioning a SAT methodology into military tools.

Understanding that scaffolding may reach an enhancement threshold as content complexity increases, ISR SMEs came together to discuss the avail- able resources and to develop a SAT hybrid-approach. This resulted in the development of the Methods for Defining Analytical Questions (MDAQ). MDAQ supports operators in assessing, processing, hypothesizing, and deliv- ering a decision based upon EEI detection. In this study, participants in the MDAQ group had the opportunity to repeat the workflow process and iden- tify up to four EEIs and their relational association regarding a possible solution. As a result, they were able to correctly identify more EEIs com- pared to both scaffolding and control. Although, this did not translate into higher performance accuracy. However, after analyzing the open-field text responses, it was discovered that only 5 of the 25 participants attempted to repeat the workflow process four times. Of those 5 participants, 4 of them arrived at the correct solution (80%). As there was no mandate requiring the participants to complete the workflow process all four times, future research needs to be conducted requiring participants to repeat the process to deter- mine if repetition enhances critical and rational thinking resulted in improved performance.

Nevertheless, the findings from this study provided a foundational baseline that structured analytic techniques (SATs) can improve information process- ing during analytical decomposition of an unstructured narrative. The USAF is moving in the right direction by evaluating the efficacy of SATs and how the methodologies can be transitioned into military environments for current and future tool development. In doing so, our warfighters and strategic planner will be able to improve human factor decision-making in near-real time.

Evaluating the Efficacy of Structured Analytic Techniques (SATs) 247

ACKNOWLEDGMENT

The views, opinions, and/or findings contained in this paper are those of the author and should not be interpreted as representing the official views or policies, either expressed or implied, of the Air Force, Air Force Research Laboratory or the Department of Defense.

REFERENCES

Belland, B., Walker, A., and Kim, N., 2017. A bayesian network meta-analysis to synthesize the influence of contexts of scaffolding use on cognitive out- comes in STEM education. Review of Educational Research, 87(6), 1042–1081. doi.org/10.3102/0034654317723009 Cann, D., Goelzhauser, G., and Johnson, K., 2014. Analyzing text com- plexity in political science research. Political Science & Politics, 663–666. doi.org/10.1017/s1049096514000808 Chang, W., Berdini, E., Mandel, D., and Tetlock, P., 2017. Restructuring struc-

tured analytic techniques in intelligence. Intelligence and National Security, 33, 337–356. doi.org/10.1080/02684527.2017.1400230 DeWiggins, S., Hite, B., and Alston, V., 2010. Personal performance plan: Application of mental skills training to real-world military tasks. Journal of Applied Sport Psychology, 22(4), 458–473. doi.org/10.1080/10413200.2010.500606 Ge, X., and Land, S., 2003. Scaffolding students’ problem-solving processes in an ill-

structured task using question prompts and peer interactions. Educational Tech- nology Research and Development, 51(1), 21–38. doi.org/10.1007/bf02504515 Hambrick, D., and Engle, R, 2002. Effects of domain knowledge, working memory capacity, and age on cognitive performance: An investigation of the knowledge-is-power hypothesis. Cognitive Psychology, 44(4), 339–387. doi.org/10.1006/cogp.2001.0769 Javed, W., and Elmqvist, N, 2013. ExPlates: Spatializing interactive analy- sis to scaffold visual exploration. Computer Graphics Forum, 32, 441–450. doi.org/10.1111/cgf.12131 Jirasataporn, N., and Hiranburana, K., 2019. Scaffolding students’ critical thinking

reading ability through appraisal theory. 2nd International Conference on Applied Liberal Arts, 96. researchgate.net/publication/334679094 Jones, N., 2018. Critical epistemology for analysis of computing hypothesis. Intelligence and National Security, 33(2), 273–289. doi.org/10.1080/02684527.2017.1395948 Kahneman, D., 2011. Thinking, fast and slow. New York: Farrar, Straus and Giroux Martinez, D., 2010. ISR sensor processing and data exploitation. IEEE Radar Conference, 1444–1445. doi.org/10.1109/radar.2010.5494390 Nelson, J., McKinley, R., Phillips, C., McIntire, L., Goodyear, C., Kreiner, A., and

Monforton, L., 2016. The effects of transcranial direct current stimulation (tDCS) on multitasking throughput capacity. Frontiers in Human Neuroscience, 10, 589. doi.org/10.3389/fnhum.2016.00589 Omohundro, S., 2014. Autonomous technology and the greater human good.

Journal of Experimental & Theoretical Artificial Intelligence, 26(3), 303–315. doi.org/10.1080/0952813x.2014.895111 Pherson, R., and Heuer, R., 2020. Structured analytic techniques for intelligence analysis. CQ Press.

Vogel-Walcutt, J., Gebrim, J., Bowers, C., Carper, T., and Nicholson, D., 2011.

Cognitive load theory vs constructivist approaches: Which best leads to effi- cient, deep learning. Journal of Computer Assisted Learning, 27(2), 133–145. doi.org/10.1111/j.1365–2729.2010.00381.x Walsh, T., and Volsko, T., 2008. Readability assessment of internet-based consumer health information. Respiratory Care, 53(10), 1310–1315. rc.rcjournal.com/content/53/10/1310 Yelland, N., and Masters, J., 2007. Rethinking scaffolding in the infor- mation age. Computers & Education, 48(3), 362–382. doi.org/10.1016/ j.compedu.2005.01.010

Cover of the proceedings volume Augmented, Virtual and Mixed Reality Simulation

Explore the complete digital volume

Augmented, Virtual and Mixed Reality Simulation

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-94-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →