Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Motion Capture Body Tracking and Functional Safety in Dynamically Controlled Theatre Automation Systems

University of Wisconsin – Madison, WI, 53706, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1004077

Abstract

Live theatrical performance is an ever-evolving art form in which visionary theater makers are incorporating evolving technologies into performances to connect and engage modern audiences. Recent developments in theatrical motion control systems are enabling vibrant and adaptive control through dynamic automation. Traditional theatrical motion control systems use a set path motion profile to produce predictable movements of scenery and people through space and time. New dynamic control systems utilize an external generated set point parameter to specify the desired motion. This flexibility enables a DJ to control the movement of flown digital chandeliers above a dance floor with their regular beat control or for Alice to control her flown decent as she tumbles down the rabbit hole by changing the position of her arms. Putting the control of the equipment into the hands of the performers is a significant leap in evolution stage automation control.To accomplish the motion tracking of performers on stage, the research team developed a unique set-up of Microsoft Kinect sensors to monitor the stage environment. The skeletal models of up to four performers were captured and transmitted via an ethernet network to the video game engine. The main benefit of this tracking method is that it allowed for both traditionally costumed performers and regularly dressed audience members to be captured by the system. Most motion capture systems in entertainment require the use of custom body suits which can disenchant and distract audiences. Our system maintains the traditional relationship between costume and character while collecting the necessary interactivity data.The natively 3D nature of the video game environment is optimally suited to handle the interaction between performer and the digital environment. Using the skeletal tracking models from the sensor system, the video game engine allowed the performers to dynamically interactive with the stage environment. They could open doors, swat away birds, and flap their wings to fly away. The added benefit of the system was that it allowed members of youth young audiences to be called onto stage to further the narrative with their actions. The resulting system enables a new performance methodology with exciting new options for theatrical storytelling, educational training, and interactive entertainment. This evolution of control adds risks beyond those present in deterministic systems.Dynamic control of entertainment automation systems requires strong conformance to functional safety principles to mitigate the risks to affected personal and environments. Functional Safety is the mitigation of unacceptable injury risks through the implementation of one or more automatic protection functions (often called safety functions). In traditional theatre environments, safety functions have historically been limited to “Emergency Stop” functions which halt all motion when an operator presses “the big red button”. As systems become more complex and utilize dynamic control, theatrical automation control systems need to utilize a functional safety evaluation process to better protect persons and the environment for systematic and random failures in the systems. Industry experts develop best practice procedures to address safety concerns which are written into documents called standards.The presenter conducts research at the intersection of entertainment and engineering, has actively participated in the development of numerous American national standards with the Technical Standards Program of ESTA (Entertainment Services and Technology Association), and is a dual certified functional safety engineer (TÜV Rheinland and Underwriters Laboratories). This presentation will go through the functional safety considerations required to implement dynamic control per national and international standards using examples from realized performance projects led by the presenter.

INTRODUCTION

Traditional theatrical motion control systems use a set path motion profile to produce predictable movements of scenery and people through space and time. New dynamic control systems utilize an external generated set point to specify the desired motion. This flexibility enables a DJ to control the movement of flown digital chandeliers above a dance floor with their regular beat control or for Alice to control her flown decent as she tumbles down the rabbit hole by changing the position of her arms. Putting the control of the equipment into the hands of the performers is a significant leap in evolution stage automation control.

THE ALICE PROJECT The technology behind most live theatrical performance events has been standardized into multiple entertainment control systems. Traditional per- formance practice dictates that each of these systems has an operator to run it, and this work is in conjunction with, though independent of, the onstage performer. The performers’ movement either influences the operation of these technology systems or is driven by the output of the technology of these sys- tems. For example, a spotlight operator follows the movement of a dancer, while an actor adjusts their speaking tempo to sync with a recorded video. The philosophy of our Augmented Live Interactively Controlled Environment (ALICE) Project is to place the control of these systems with the performer.

© 2023. Published by AHFE Open Access. All rights reserved. 705

706 Lisowski

This dynamic shift was developed and tested through multiple performances in the spring of 2014 at the University of Wisconsin – Madison.

The ALICE Project is a multidisciplinary interactive production method- ology that melds traditional theatrical production disciplines with emerging technologies. The ALICE Project enables the performer (i.e., actor, dancer, musician, etc.) to simultaneously interact with and control multiple aspects of a dynamic stage environment (Figure 1). By integrating video projection, motion control, motion capture, a video game engine, and virtual reality technologies together, the project enables new possibilities in live perfor- mance that enhance the experiences of both the performer and the audience (Lisowski et al. 2023).

Figure 1: ALICE stage set-up (Daniel T. Lisowski, 2023).
Original PDF Figure 1:

MOTION CAPTURE To accomplish the motion tracking of performers on stage, the research team developed a unique set-up of Microsoft Kinect sensors to monitor the stage environment. The skeletal models of up to four performers were captured and transmitted via an ethernet network to the video game engine. The main benefit of this tracking method is that it allowed for both traditionally cos- tumed performers and regularly dressed audience members to be captured by the system. Most motion capture systems in entertainment require the use of custom body suits which can disenchant and distract audiences. Our system maintains the traditional relationship between costume and character while collecting the necessary interactivity data.

Motion Capture Body Tracking and Functional Safety 707

Figure 2: ALICE control system diagram (Daniel T. Lisowski, 2023).
Original PDF Figure 2:

Two Kinect sensor assemblies were used to track the motion of the performers in the stage environment. These assemblies consisted of one Microsoft Kinect V.2 sensor and one computer that meet the minimum requirements for Kinect USB3.0 compatibility. A project specific application captured the joint position tracking data from the Kinect sensors and trans- mitted this information to the two cooperating systems: Unity video game engine and the automation motion control system.

UNITY VIDEO GAME ENGINE The natively 3D nature of the video game environment is optimally suited to handle the interaction between performer and the digital environment. Using the skeletal tracking models from the sensor system, the video game engine allowed the performers to dynamically interactive with the stage envi- ronment. They could open doors, swat away birds, and flap their wings to fly away. The added benefit of the system was that it allowed young audi- ence members to be called onto stage to further the narrative with their actions. The resulting system enables a new performance methodology with exciting new options for theatrical storytelling, educational training, and interactive entertainment (Lisowski et al. 2023). While exciting, this evo- lution in control systems adds risks beyond those present in deterministic systems.

708 Lisowski

DYNAMIC CONTROL Dynamic control of entertainment automation systems requires strong con- formance to functional safety principles to mitigate the risks to affected personal and environments. Functional Safety is the mitigation of unac- ceptable injury risks through the implementation of one or more automatic protection functions (often called safety functions). In traditional theatre environments, safety functions have historically been limited to “Emergency Stop” functions which halt all motion when an operator presses “the big red button”. As systems become more complex and/or utilize dynamic control, theatrical automation control systems need to utilize a functional safety eval- uation process to better protect persons and the environment for systematic and random failures in the systems. Industry experts develop best practice procedures to address safety concerns which are written into documents called standards.

Industry standards provides minimum design specifications to mitigate many of the typical hazards associated with performer flying (ESTA E1.43 2016). For instance, redundant load securing devices (i.e. brakes) must be installed on any machine utilized in performing flying effects to eliminate a single brake failure leading to a hazardous event. The typical theatrical performer flying hoist used in a cue-based playback system has its required minimum specifications well defined in the standard. When using dynamic control systems, the uncertainty of motion presents additional hazards to these already complicated systems.

The presenter conducts research at the intersection of entertainment and engineering, has actively participated in the development of numerous Amer- ican national standards with the Technical Standards Program of ESTA (Entertainment Services and Technology Association), and is a dual certified functional safety engineer (TÜV Rheinland and Underwriters Laboratories). In accordance with the ANSI E1.43 – Performer Flying Standard, a full Risk Assessment/Risk Reduction process was implemented throughout the design, fabrication, installation, and operation phases of a project. Throughout the process, the safety of the performer is paramount. While the process of this assessment might be interesting for some readers, this information will be omitted from the paper and reserved for a future publication. The resulting required safety functions from the assessment are presented in this paper.

SAFETY FUNCTIONS Safety functions are implemented by the functional safety system to achieve or maintain a safe state in respect of a specific hazardous event (adapted from IEC 61508-4, 2010). In the ALICE system, these safety functions are programmed into a safety control module in the Beckhoff Ethercat terminal system, EL6910. This terminal and the associated software conforms to EN ISO 13849-1:2015 (Cat. 4, PL e) and IEC 61508:2010 (SIL 3) (Beckhoff, 2023). The servo drive used to power the motor in our performer flying hoist was a Beckhoff AX5140-0000-0200 with the TwinSAFE drive option card AX5805. This combination of motor, drive, and safety card allows for the

Motion Capture Body Tracking and Functional Safety 709

implementation of numerous safety function as defined in IEC 61800-5-2 “Adjustable speed electrical power drive systems – Part 5-2: Safety require- ments – Functional”. The following safety functions were implemented in the ALICE dynamically controlled performer flying hoist.

Stopping Functions (STO, SS1, & SS2) A variety of stopping functions are defined in IEC 61800-5-2 for use by the functional safety system to quickly and securely bring motion to a halt. The three stopping functions specified in this standard used in the ALICE system were Safe Torque Off (STO), Safe Stop 1 (SS1), and Safe Stop 2 (SS2).

Safe Torque Off (STO) is the most abrupt of the stopping functions and attempts to bring motion to a halt as quickly as possible by immediately removing power to the actuator/motor and engaging load securing devices (i.e. brakes). This corresponds to an uncontrolled stop in accordance with stop category 0 of IEC 60204-1. The rapid application of brakes can impart substantially negative acceleration on the performer depending on the speed when the stopping function is first implemented. ANSI E1.43 specifies the maximum acceleration forces applied to a performer for multiple performer orientations. The ALICE hoist was specifically designed to limit these forces to allowable limits for the “Eyeballs Down” (+Gz) orientation.

Safe Stop 1 (SS1) is a stopping function which initiates and controls the motor deceleration for a specific time interval or to a specific set speed before initiating a STO stopping function. This corresponds to a controlled stop in accordance with stop category 1 of IEC 60204-1. The controlled deceleration implemented by this stopping function reduces the shock imparted on the performer and machinery.

SS2 is a stopping function with initiates and controls the motor decelera- tion rate to stop the motor and maintains power available at the motor. This stopping function is typically not appropriate for safety critical applications and was used to limit the unintended direction of travel of the performer.

Safely-Limited Speed (SLS) The SLS function prevents the motor from exceeding the specified speed limit (61800-5-2, 2007). As the motor speed is directly proportional to the performer speed, this function was added to ensure that the perform is not traveling too fast as they approach the ends of travel. 3 ft/s (feet per second) was determined by the risk assessment to be safe speeds while approaching the upper physical limit of travel (and the support structure) and the lower limit of travel (the stage floor). If the monitored speed was exceeded with the function initiated by a mechanical limit switch, the system would initiate a Safe Stop 1 (SS1) stopping function.

Safely-Limited Position (SLP) The SLP function prevents the motor shaft from exceeding the specified position limits (61800-5-2, 2007). As the rotational position is directly pro- portional to the performers vertical position, this function was specifically added to limit the upward vertical travel to minimize the risk associated

710 Lisowski

with the performer contacting the structural supports. As striking the upper support structure provided the most catastrophic hazard in the system, this additional precaution was implemented to minimalize to occurrence of this hazard. Upon activation, the SLP function would implement an STO stopping function.

Safe Direction (SDI) The SDI function prevents motor shaft from moving in the unintended direc- tion (61800-5-2, 2007). The ALICE system utilized a subset of SDI to limit motor shaft rotation in the positive direction on movement called SDIp (p for positive). This SDIp function was coupled with the same mechanical limit switch used for the “Normal Limit” in performer flying systems. This additional protection was implemented to further minimize the possibility of the hoist continuing upward movement upon reinitialization of the control signal. SDI was implemented with a SS2 stopping function.

Safely-Limited Acceleration (SLA) The SLA function prevents the motor from exceeding the specified accelera- tion limit (61800-5-2, 2007). This function was implemented with substan- tially different values for positive and negative accelerations. The positive acceleration limit was set to three times the acceleration due to gravity, result- ing in a maximum upward acceleration force of three G’s. This acceleration was necessary to represent the absurdity of “falling down the rabbit hole” in a vertically constricted space. The maximum negative acceleration was lim- ited to 0.9 times the acceleration due to gravity. This safety measure ensured that the hoist would not spool out cable faster than gravity was pulling the performer down resulting in an uncontrolled fall from height or shock from re-engaging a slack lift line. The negative SLA limit mitigated likelihood of both of these hazard to an acceptable level. The SLA function executed a STO stopping function when either limit was exceeded. (fpn: Beckhoff uses Safe Maximum Acceleration (SMA) in their production documentation.)

Additional Safety Functions A few additional safety functions were implemented which are typical for most performer flying systems. Per E1.43, all performer flying systems shall include both an Emergency Stop button and Ultimate Limits at each end of travel to immediately implement a STO stopping function. A “hold-to-run” enable button was added so that an operator could halt motion using a SS1 stopping function by simply removing their finger. This feature was added during the risk assessment process to deal with the hazardous instance when another person entered the stage area during a performance. If this person were to be contacted by the flown perform, medical treatment was fore- seeable result. As the risk assessment determined that the likelihood of this occurrence was Moderate, the scoring determined that “hold-to-run” enable button would be sufficient to mitigate the risks associated with this hazard if the button was run through the functional safety system.

Motion Capture Body Tracking and Functional Safety 711

The validity of data was determined to be a potential fault that could lead to a hazardous situation. Comparison measures were implemented in the con- trol system to check that the data received by the motion control system was valid and congruent prior to any motion was initiated. Any incongruencies in the data would initiate a system fault, halting all motion with a SS1 stopping function.

SYSTEM LIMIT COMPARISON ANSI E1.43 – Performer Flying Systems requires that automated performer flying systems contain multiple control and safety measures to minimize the risk of harm to the performer and others. In relation to operational and functional safety, the standard requires that automated systems contain three “limits” which restrict further motion in the associate direction of travel: “Soft”, “Normal”, and “Ultimate”. Soft limits, which are coded into the software parameters, are set just outside the normal operating range for the machine. When exceeded, these soft limits initiate a controlled stop via the motion controller and allow motion only in the opposite direction of travel. Normal limits are physical switches placed just beyond the Soft limits which initiates a controlled stop via the motion control system and allows opposite direction of motion, similar to the Soft limits. Unlike the previous two limits, Ultimate limits are part of the functional safety system. Once these mechani- cal switches are activated, a STO stopping function is initiated and motion in all directions is halted. These three limits are typical of all performer flying hoists, including the ALICE performer flyer hoist. In addition to these stan- dard operational and safety functions, the Risk Assessment determined that the previously described safety functions be implemented for this application. The location of these devices are indicated in Figure 3.

Figure 3: Limits in ALICE performer flying system (Daniel T. Lisowski, 2023).
Original PDF Figure 3:

712 Lisowski

CONCLUSION

Motion capture systems incorporated into live perform provide theatre mak- ers with new options in storytelling. The data collected by these systems can be used to dynamically drive the projected virtual world and stage automa- tion systems. This dynamic control element brings new and increased risks to the performer in a controlled performer flying system. The implementation of safety functions in the functional safety system can mitigate these risks to acceptable levels.

ACKNOWLEDGMENT

Support for this research was provided by the Office of the Vice Chancel- lor for Research and Graduate Education at the University of Wisconsin– Madison with funding from the Wisconsin Alumni Research Foundation (WARF), the Grand Challenges Engage Grant provided School of Educa- tion at the University of Wisconsin – Madison, and the Impact 2030 School of Education Faculty Fellowship program at the University of Wisconsin – Madison.

REFERENCES

ANSI/ASSE Z690.1–2011. (2011). Vocabulary for Risk Management. Washington, D. C.: American National Standards Institute. ANSI/ASSE Z690.2–2011. (2011). Risk Management Principles and Guidelines.

Washington, D. C.: American National Standards Institute. ANSI/ASSE Z690.3–2011. (2011). Risk Assessment Techniques. Washington, D. C.: American National Standards Institute. Bauer, Michael. (2002) “Functional Safety of a Theatre Stage Machinery Control

System” proceedings from the Seventh Australian Workshop on Safety-Related Programmable Systems, SCS2002, Adelaide, Australia. Beckhoff (May15, 2023) EL6910 | EtherCAT Terminal communication interface, TwinSAFE Logic. Beckhoff website: https://www.beckhoff.com/en-us/products/au tomation/twinsafe/twinsafe-hardware/el6910.html Carroll, L. (1998) Alice’s Adventures in Wonderland. VolumeOne Publishing,

Chicago. ESTA TSP: Entertainment Technology – Performer Flying Systems, ANSI E1.43 –

2016. IEC: Adjustable speed electrical power drive systems – Part 5-2: Safety requirements – Functional, IEC 61800-5-2:2007. IEC: Functional Safety of Electrical/Electronic/Programmable Electronic Safety-

Related Systems. Parts 1-7: 2010. Lee, J., Kim, Y., Heo, M.-H., Kim, D., Shin, B.-S. (2015) Real-time projectionbased augmented reality system for dynamic objects in the performing arts. Symmetry. 7(1), 182–192. Lisowski, D., Ponto, K., Fan, S., Probst, C., Sprecher, B. (2023) “Augmented Reality

into Live Theatrical Performance.” In Springer Handbook of Augmented Reality, pp. 433–450. Springer, Cham. Marner, M. R., Smith, R. T., Walsh, J., Thomas, B. H. (2014) Spatial user interfaces for large-scale projector-based augmented reality. IEEE Comput. Graph. Appl. 34(6), pp. 74–82.

Motion Capture Body Tracking and Functional Safety 713

Mavridis, N., Hanson, D. (2009) “The IbnSina center: An augmented reality theater with intelligent robotic and virtual characters”. The 18th IEEE International Symposium on Robot and Human Interactive Communication. RO-MAN 2009, pp. 681–686. IEEE, Toyama, Japan. Meador, W. S., Rogers, T. J., O’Neal, K., Kurt, E., Cunningham, C. (2004) “Mixing

dance realities: Collaborative development of live-motion capture in a performing arts environment.” Comput. in Entertain. (CIE). 2(2), pp. 12–22. Ponto, K., Lisowski, D., and Fan, S. (2016) “Designing extreme 3D user interfaces for augmented live performances,” 2016 IEEE Symposium on 3D User Interfaces (3DUI), Greenville, SC, USA, pp. 169–172. Sparacino, F., Davenport, G., Pentland, A. (2000) “Media in performance: Interactive

spaces for dance, theater, circus, and museum exhibits.” IBM Syst. J. 39(3–4), pp. 479–510. Sparacino, F., Wren, C., Davenport, G., Pentland, A. (1999) “Augmented performance in dance and theater.” Int. Dance Technol. 99, pp. 25–28. Weijdom, J. (2016) “Mixed reality and the theatre of the future.” in: IETM Spring

Plenary Meeting.

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-87-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →