INTRODUCTION
Extended reality (XR1), head-mounted displays (HMD), immersive simulators, and advanced display-control systems (for manned or unmanned vehicles) are growing in military importance and, after a lengthy delay, finally disseminating widely into the general population. Relevant systems have 1Virtual reality (VR) or augmented reality (AR).
© 2023. Published by AHFE Open Access. All rights reserved. 140
Attention Military/Commercial Simulation Developers, Users, & Trainers 141
been developed, are planned, or are already in use or readily adaptable from existing systems to augment training for pilots, astronauts, ship/submarine handlers, emergency medical responders, surgeons, and therapists (e.g., to treat phobia or post-traumatic stress). XR, HMD, and other immersive technologies are also in development to enhance decision-making during interaction with complex transportation, teleoperation, or medical systems within the military, aerospace, or industrial domains.
One of the key reasons vision-centric displays such as XR, HMD, and immersive simulators did not disseminate more rapidly was because of the noxious motion-sickness-like symptoms they elicited (Stanney et al., 2020), e.g., stomach symptoms, dizziness, headache, and visual/eyestrain symptoms collectively referred to as visually-induced motion sickness (VIMS) (Cha et al., 2021). VIMS is considered a sub-type of motion sickness because: a) the symptoms are similar, b) whole-field visual motion affects some of the same vestibular brain centers as real motion, and c) the sensorimotor challenges are comparable concerning the way simulated environments are displayed to the senses versus normal visual-vestibular-tactile-kinesthetic integration as it occurs in the natural world. (See Keshavarz et al., 2014; Lawson, 2014a; Cha et al., 2021; Gavgani et al., 2018.)
VIMS-consistent symptoms have received widespread notice recently, having halted a major military acquisition of AR “combat goggles”(designed to support situation awareness and decision-making) (McAuliffe, 2023). Recent and ongoing work by a NATO Human Factors in Medicine committee (NATO Science & Technology Office, 2021) is studying the causes of these ill effects and seeking to understand who is most vulnerable. The answer to these questions is important to system design tailoring and targeted interventions for military training and decision aids, as well as for long-term commercial success and the avoidance of developer/trainer liability (which could arise due to a flawed product making some users sick, contributing to accidents, or harming career prospects).
A commonly published assertion concerning individual differences in visual/motion sickness susceptibility concerns the contributing role of biological sex, with some studies and several literature reviews claiming that the preponderance of evidence indicates that women2 are far more susceptible than men to motion sickness or VIMS induced by real motion, simulators, XR, HMD, or combinations of these stimuli. One article stated in its title that one type of VR is “sexist3 in its effects” (Munafo et al., 2017, p. 889). Some debate arose after this article, as a subsequent study identified non-sex- specific reasons for the difference which could be fixed readily by improving display ergonomics, with its authors concluding that VR is “sexist but it does not have to be” (Stanney et al., 2020, p. 1). Similarly, a review by Grassini and Lauman (2020) questioned the basic premise of sex difference, asking: “Is 2“Women” refers to the common birth case of having no Y chromosome present among a complement of 46 chromosomes, and is not intended to generalize to other genetic birth cases, to sex reassignment, or to gender identification. 3We quote the word “sexist” to help researchers link this paper to the debate among the other three publications (wherein the term has been coopted by the cited authors), because we think the present paper sheds new light on that debate.
142 Lawson and Bolkhovsky
there sufficient evidence for a gender imbalance to define such technology as ‘sexist’?”
The present study expands upon this debate in the literature by evaluating a larger sample of relevant literature than prior studies, and does so across a wider range of stimuli, to determine the accuracy of the assertion that most literature supports the hypothesis that women are more susceptible to visual/motion sickness. This question is important to answer, as women are the largest demographic subgroup for whom increased susceptibility has been posited, they represent up to 50% of the potential pool of job or military recruits, and they are traditionally a demographic group subjected to discrimination. Therefore, we have evaluated the amassed literature.
The motivation for investigating the literature concerning this issue began circa 2002, after a group of scientists met with an Office of Naval Research representative and most of them opined that purported sex differences had been verified by most studies of motion sickness/VIMS. To determine whether such advice was fully justified, an internal literature review was initiated, and it has been updated periodically since then. This paper represents our first dedicated publication fully describing and updating the findings from these reviews. The following sections briefly describe the approach taken to review this question over the years, and the inferences drawn.
APPROACH Literature searches were completed on four occasions: during the Summers of 2003 and 2014, the Spring of 2021, and the Winter of 2022. The searches exploited ∼20 literature sources or search engines which varied over the years as new tools became available.4 Searches were made for direct empirical/survey studies including such terms as: [“sex,” “women,” “female,” or “gen-der”5] + [“virtual,” “augmented,” “simulator,” or “motion”] + [“sickness” or “cybersickness”].
Studies were included if they clearly provided quantitative findings from direct empirical laboratory, field, or survey research, while reviews or anecdo- tal observations were excluded. The inclusion criteria were otherwise liberal (e.g., they included any non-redundant dissertations or government technical reports obtained) because the intent was to be comprehensive and to mini- mize “file drawer” bias caused by the fact that studies with negative findings tend to be accepted less often by journals (Pautasso, 2010), or to be less accessible/cited.
The objective of these reviews was to establish whether a compelling majority of studies yield results that clearly support the hypothesis that women or more susceptible to motion/visual sickness. Studies were not
4The original 2003 systematic review included sources such as Medline, PubMed, EBSCO, OVID, JAMA, Lancet, NEJM, PsychLIT, Psych-Info, NTIS, DTIC, Google (Scholar was not released until Nov 2004), Soulsearch, two annotated bibliographies, and three hardcopy archives. Recent updates relied more upon Scholar, DTIC, and literature alert systems, e.g., Semantic Scholar. 5“Gender” is not recommended by APA for referring to biological sex, but still appears in that usage. It is not a critical search term for locating recent publications but was useful for finding older ones (and for searching for findings inside old/new publications).
Attention Military/Commercial Simulation Developers, Users, & Trainers 143
limited to XR but included other sources of relevant visual and/or physical motion also (e.g., optokinetic drums, flying or driving simulators, vehicle transportation situations). This wide review scope was justified for several reasons:
- Because VIMS is merely one aspect of the overall motion sickness response;
- Because the majority of the relevant sex difference literature would be ignored by limiting consideration solely to XR;
- Because some of the more recent XR studies have employed overly weak stimuli, which could bias interpretation towards a negative finding (Lawson & Stanney, 2022);
- Because there is often no pure distinction between visual and physical motion stimuli (e.g., XR involves head/body movement, while vehicle motion happens in the presence of dynamic visual motion and processing of discordant visual stimuli);
- Because it can be important for military personnel deployed aboard moving vessels to engage in vision-centric XR/simulation training (Muth & Lawson, 2003; Cohn et al., 2003; Muth et al., 2006).
FINDINGS This section summarizes the interim findings from three preliminary literature searches that have been executed since the early 2000s, and the latest cumulative findings of all reviews, as of November 2022:
- Regarding the first 2003 review: Lawson et al. (2004) presented findings from the original systematic review at a conference, identifying 46 relevant studies from 1940-2001. They concluded that 26/46 (56.5%) of the studies were consistent with the hypothesis that women are more susceptible to motion sickness or VIMS.
- Three updates to the original review were completed since 2003: in 2014 (presented by Lawson et al., 2015), in 2021 (briefly summarized in a paper not dedicated to women’s susceptibility [Lawson et al., 2021]), and on 19 Nov 2022, as part of this first dedicated publication of these cumulative review findings-to-date. These partial literature updates identified another 30 relevant studies (above what was reported in 2004). Of these, 11/30 (36.7%) were consistent with women being more susceptible.
When the four reviews were joined into cumulative results, the authors found that 37/76 or 48.7% of all studies obtained so far are consistent with women being more susceptible. The implications of this finding are described below, as well as several confounding trends in the literature.
CONCLUSION
Implications of This Study After amassing what we believe is the largest known collection of studies relevant to sex differences in visual/motion sickness susceptibility, this first dedicated publication of our results-to-date found that only 48.7% of studies
144 Lawson and Bolkhovsky
yielded findings consistent with the hypothesis that women are more susceptible than men. This proportion falls short of a majority of the relevant studies identified, and certainly falls far short of the large proportion that would be necessary to support the conclusion that a clear and strong effect has been widely replicated under various circumstances. It is, therefore, presently required for scientists to doubt any assertion to the effect that most literature studies have found women to be more susceptible.
Wider Implications When we related our findings to other reviews, it was encouraging that our estimate of proportion did not represent an extreme outlier. In fact, our 48.7% estimate is bracketed by the conclusions drawn from recent reviews by others who employed different review methods and scopes. For example, Grassini & Laumann, 2020, concluded that 5/24 (21%) relevant studies they individually identified were consistent with women being more susceptible, while we infer from MacArthur et al.’s 2021 VR cybersickness- focused study that 33/56 (58.9%) of the relevant studies they identified were individually consistent with women being more susceptible.6 None of the three estimated proportions mentioned immediately above are sufficient to view a potential sex difference as conclusively established. Nevertheless, we continue to encounter the implication that studies generally agree expressed women are more susceptible to motion sickness and/or VIMS (expressed in group meetings, conference talks, review papers, and the introduction sections of individual research studies).
Based on the overall findings to date, we remind researchers that it is necessary to reserve their judgments on this issue until conclusive evidence has been obtained. No scientist should feel justified in concluding that a particular group is inherently more affected by a stimulus based upon approximately half of the published studies individually concluding that this is the case. This is especially true when most of the positive evidence of group difference in the literature is based upon self-report surveys rather than direct experimental comparisons which are tightly controlled for interpretation confounds. In fact, at least 19 possible confounds to interpretation exist that would need to be controlled before one can conclude any observed group difference is due to biological sex, per se (Lawson et al., 2004; 2014a).
Study Strengths and Limitations There are some strengths and limitations of the current study, which will be described below. Two major strengths of the current research are as follows:
1) The findings are drawn from the largest known sample of relevant studies in the literature.
6Among 56 relevant papers involving studies and employing men and women, we inferred that 33 found a difference (∼7 where sex was a primary focus plus 26 where sex was not, albeit the authors point out interpretation of primary studies varies widely).
Attention Military/Commercial Simulation Developers, Users, & Trainers 145
- This means that our findings are more resistant to selection bias (e.g., the “file drawer problem”) than many other reviews, due to our liberal inclusion criteria.
- Also, our large sample benefits from the inclusion of more studies where functionally significant levels of sickness were elicited, compared to the tendency for too much of the VR-specific literature to draw conclusions based on overly weak/basement stimuli sometimes yielding statistical differences in motion sickness that are of limited functional relevance (Lawson & Stanney, 2021).
- Finally, since this review is drawn from a wide array of both motion sickness and VIMS studies spanning >80 years, it is likely to afford more information for assessing potentially confounding chronological trends than would be the case for more date-constricted collections of studies. For example, Lawson et al. (2004) noticed a concerning chronological trend wherein studies done from 1940–1979 were ∼2X more likely to conclude that women were more susceptible, compared to studies done from 1980-2001. That trend continues, as shown in the Findings above, i.e., the cumulative findings from 1940–2001 were ∼1.5X more likely to conclude women were more susceptible than the studies done after 2001. This trend would not be expected if differences in male versus female susceptibility were entirely determined by genetic sex differences. Non-sex-specific contributors might include a chronological lessening of subject/experimenter biases due to changing cultural norms, greater recent female experience with challenging motion and visual motion situations (leading to beneficial adaptation), or other causes. (We are not aware of any empirical lab studies of motion sickness/VIMS that have directly manipulated sex bias; however, we have found that the top Google Image searches for “motion sickness” tend to depict women suffering from motion sickness/VIMS (or utilizing sickness cures) far more commonly than men.7)
2) Our findings are based upon what the provided, quantitative results of the obtained studies directly support, reducing the unfortunate tendency of some past presentations, reviews, or publication Introduction sections to merely report trends based upon the overall conclusions of the study authors (e.g., as stated in their Abstract or Discussion sections). Our method reduces bias in this literature review which could be caused by studies inferring a sex difference in instances where a difference was implied, but the findings failed to reach statistical significance, no quantitative data were reported, or the study involved several manipulations yielding mixed results, the minority of which supported a sex difference.
7Images of women appeared 4.5X more often than images of men during a March 2014 Google Image search for “motion sickness” (Lawson et al. 2004) and 2X more commonly during a Dec 2022 search, based on the first two-pages visible at 100% screen size (∼25-27 images). This 2-4.5X trend could not be accounted for by a tendency for women to appear more in image searches for control words such as “person” or “motion,” but there was a lesser (1.5-4X) tendency for women to appear more in “sickness.”
146 Lawson and Bolkhovsky
There are also two limitations of this literature review, which we plan to overcome in upcoming research:
1) While our review spans more than 80 years of studies, there is a five- year gap in our review that must be filled, viz., for the literature during 2014-2018. It should be noted, however, that a study of Institute of Electrical and Electronics Engineers (IEEE) VR conferences findings from 2015–2019 (Peck et al., 2020), which covered most of our gap period, aligned with our study, concluding that women’s presumed susceptibility to cybersickness may simply be due to biased study design/sampling (e.g., the studies having a lower proportion of women participants tending to observe a sex difference). 2) While assessing scientific consensus among the results of published studies by different groups is central to the scientific process, the scientific literature is not a democracy, wherein a simple tally of the studies in favor of a hypothesis definitively decides an issue, because the scientific quality of each study is also important. Therefore, a consideration of the sex proportion obtained in the subset of studies of highest quality will be obtained in future, for comparison. However, readers should note that the last time study quality was checked (Lawson et al., 2014), it was found that the controlled, empirical laboratory studies directly comparing men and women while inducing motion sickness were only ∼1/3 as likely to conclude there was a sex difference, compared to the numerous self-report survey studies. We expect this trend to continue when we update the study quality literature analysis.
Study quality can be incorporated into meta-analysis, but high-quality information for meta-analysis is limited currently (MacArthur, 2021), as there are numerous confounds to interpretation (Lawson, 2014a) and no large body of tightly-controlled, direct empirical laboratory comparisons employing comparable measures and designs. Therefore, the aim of future research should be to conduct several tightly-controlled studies and incorporate the resulting findings into further meta-analyses. Such controlled lab studies are rare because they are very difficult to recruit for and execute. Nevertheless, interest in, and support of such studies should increase because increasing numbers of women are entering all types of military and industrial jobs. Also, there is greater emphasis on evidence-based unbiased testing and selection than in the past, and there is growing interest in individual differences research. It is possible that when enough controlled findings are obtained and pooled, women will be found to be more susceptible than men. However, we doubt the effect will be large, given the trends so far in our large review, in the aforementioned review of 24 relevant studies (Grassini & Lauman, 2020), and in two meta-analyses of the limited-quality data available so far.8
8Our paper, Grassini & Lauman, and MacArthur et al. all questioned the difference (and study quality). Saredakis et al. (2020) did not observe a difference in indirect meta-analysis of 51 relevant studies. The only interesting effect came from meta-analysis of 43 studies (Howard & van Zandt, 2021) -- a small-to-medium effect, depending upon published criteria one adopts.
Attention Military/Commercial Simulation Developers, Users, & Trainers 147
Presently, researchers are required to reserve judgment, pending a sufficient body of tightly controlled empirical studies yielding consistent findings. Firm conclusions of positive sex difference are not scientifically warranted, and premature conclusions that we have seen in publications and presentations could needlessly reduce military/workforce readiness, career prospects of women, and commercial dissemination of useful technologies. While sex may ultimately be proven to account for enough variance in susceptibility to be of interest to researchers, military/industrial training personnel seeking practical and fair methods to identify and help the most susceptible trainees in the near-term should concentrate primarily upon whether such trainees have been directly observed as highly susceptible during past exposures to the same stimulus, or failing such information, upon whether trainees and their close relatives show extreme susceptibility to relevant stimuli (Lawson, 2014a,b; Lawson et al., 2021). Karl Popper rightly taught scientists that it is necessary to doubt until doubt is no longer warranted. Such doubt is especially required when incorrect conclusions could harm the world’s largest demographic subgroup.
ACKNOWLEDGEMENT
The authors thank the following colleagues for lively discussions of potential sex differences in this area of inquiry: John Golding, Steven Kass, Kay Stanney, J. Christopher Brill, Jelte Bos, and the late Robert S. Kennedy.
DISCLAIMERS The views expressed in this report are those of the authors and do not necessarily reflect the official policy or position of the Department of the Navy, Department of Defense, nor the US. Government. The authors are employees of the US. Government. This work was prepared as part of their official duties. Title 17 US. C. 1O5 provides that ‘Copyright protection under this title is not available for any work of the United States Government.’
REFERENCES
Cha, Y. H., Golding, J. F., Keshavarz, B., Furman, J., Kim, J. S., Lopez-Escamez, J. A.,... & Lawson, B. D. (2021). Motion sickness diagnostic criteria: consensus document of the classification committee of the Bárány society. Journal of Vestibular Research, 31(5), 327–344. Cohn, J., Muth, E., Schmorrow, D., Brendley, K., & Hillson, R. (2003). Reducing
negative effects from virtual environments: Implications for just-in-time training. Proceedings, RTO HFM Symposium on Spatial Disorientation in Military Vehicles: Causes, Consequences and Cures, La Coruña, Spain, 15–17 April 2002; published in RTO-MP-086. Gavgani, A. M, Walker, F. R., Hodgson, D. M., & Nalivaiko, E. (2018). A comparative study of cybersickness during exposure to virtual reality and “classic” motion sickness: are they different?. Journal of Applied Physiology, 125(6), 1670–1680. Grassini, S., & Laumann, K. (2020). Are modern head-mounted displays sexist? A systematic review on gender differences in HMD-mediated virtual reality. Frontiers in psychology, 11, Article 1604, 1–15.
148 Lawson and Bolkhovsky
Howard, M. C., & Van Zandt, E. C. (2021). A meta-analysis of the virtual reality problem: Unequal effects of virtual reality sickness across individual differences. Virtual Reality, 25(4), 1221-1246. Keshavarz, B., Hecht, H., & Lawson, B. D. (2014). Visually induced motion
sickness—Causes, Characteristics, and Counter measures. In K. M. Stanney & K. Hale (Eds.), Handbook of virtual environments: Design, implementation, and applications (2nd ed., pp. 647–697). New York, NY: CRC Press. Lawson, B. D. (2014a). Motion sickness symptomatology and origins. In K. S.
Hale & K. M. Stanney (Eds.), Handbook of virtual environments: Design, implementation, and applications (2nd ed., pp. 531–539). New York, NY: CRC Press. Lawson, B. D. (2014b). Motion sickness scaling. In K. S. Hale & K. M. Stanney (Eds.), Handbook of virtual environments: Design, implementation, and applications (2nd ed., pp. 601–626). New York, NY: CRC Press. Lawson, B. D., Kass, S. J., Lambert, C., & Smith, S. (2004). Survey and review
concerning evidence for gender differences in motion susceptibility. Proceedings, 75th Aerospace Medical Association Annual Scientific Meeting, Anchorage, AK. Aviation, Space and Environmental Medicine, 75(4), (Suppl. 2), 105. Lawson, B. D., Proietti, P., Burov, O., Sjölund, P., Rodabaugh, T., Kirollos, R., & Bloch, M. (2021). Factors impacting cybersickness. In: Guidelines for Mitigating Cybersickness in Virtual Reality Systems. Peer-Reviewed Final Report of the Human Factors and Medicine Panel/Modeling & Simulations Group, Activity Number 323, NATO STO-TR-HFM-MSG-323, Chapter 5, 1–44. Lawson, B. D., Thompson, L. B. I., Brill, J. C., & Rupert, A. H. (2015). Two
frequently-mentioned contributors to motion sickness severity that lack sufficient proof. Proceedings, 86th Aerospace Medical Association Annual Scientific Meeting, Orlando, FL. Aviation, Space and Environmental Medicine, 86(3), 311. Lawson, B. D., & Stanney, K. M. (2021). Cybersickness in virtual reality and augmented reality. Frontiers in Virtual Reality, Oct 2021(2) 131., p. 1-4, Article 759682. MacArthur, C., Grinberg, A., Harley, D., & Hancock, M. (2021, May). You’re
making me sick: A systematic review of how virtual reality research considers gender & cybersickness. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1-15. McAuliffe, Z. (2023, Jan 12). Congress halts purchase of more Microsoft combat goggles, report says. CNET (Computer Network). Retrieved from https://www.cn et.com/ Munafo, J., Diedrick, M., & Stoffregen, T. A. (2017). The virtual reality head-
mounted display Oculus Rift induces motion sickness and is sexist in its effects. Exp Brain Res, 235(3), 889–901. Muth, E. R., & Lawson, B. (2003). Using flight simulators aboard ships: Human side effects of an optimal scenario with smooth seas. Aviation, space, and environmental medicine, 74(5), 497-505. Muth, E. R., Walker, A. D., & Fiorello, M. (2006). Effects of uncoupled motion on
performance. Human factors, 48(3), 600–607. NATO Science and Technology Office. (2021). Guidelines for Mitigating Cybersickness in Virtual Reality Systems. Peer-reviewed Final Report of the Human Factors and Medicine Panel/Modeling & Simulations Group, Activity Number 323 (NATO STO-TR-HFM-MSG-323). Pautasso, M. (2010). Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics, 85(1), 193–202.
Attention Military/Commercial Simulation Developers, Users, & Trainers 149
Peck, T. C., Sockol, L. E., & Hancock, S. M. (2020). Mind the gap: the underrepresentation of female participants and authors in virtual reality research. IEEE Trans Vis Comput Graph, 26(5), 1945–1954. Saredakis, D., Szpak, A., Birckhead, B., Keage, H. A., Rizzo, A., & Loetscher,
T. (2020). Factors associated with virtual reality sickness in head-mounted displays: a systematic review and meta-analysis. Front. Hum. Neurosci. 14:96. doi: 10.3389/fnhum.2020.00096 Stanney, K., Fidopiastis, C., & Foster, L. (2020). Virtual reality is sexist: but it does not have to be. Frontiers in Robotics and AI (7), Article 4, 1–19. Stanney, K., Lawson, B. D., Rokers, B., Dennison, M., Fidopiastis, C., Stoffregen,
T., Weech, S., & Fulvio, J. M. (2020). Identifying causes of and solutions for cybersickness in immersive technology: reformulation of a research and development agenda. Taylor and Francis, 36(19), 1783–1803.