Gordon Carlson, Honesty Beaton (2025). Realtime Video Underlay for Accessible Television Graphics. In: Human Factors and Systems Interaction. AHFE Open Access, vol. 192, pp. 50–57. https://doi.org/10.54941/ahfe1006627
Realtime graphics in televised sports have become a staple of the genre. While impressive and effective, existing tools for integrating 3D graphics into televised playing areas are complicated and expensive, limiting their use to only a handful of large-scale television productions. Existing hardware-based solutions are costly and offer limited flexibility. This paper presents a software-based approach for implementing realtime sports graphics on consumer-grade equipment, potentially serving a range of stakeholders who would benefit from the technology but do not have the requisite infrastructure or financial means. Using a combination of open source and widely supported commercial tools, Realtime Video Underlay instead leverages a new approach to tracking physical objects in space and controlling realtime graphics by adapting concepts from augmented reality software and integrating with existing television production and streaming platforms. The results of this research show the potential of software-based methods as a key contender for accessible sports broadcasting for groups with limited resources or technical knowledge. The paper outlines the key components and design of the tool with a specific case study, identifies strengths and weaknesses, and provides an overview of the next version of the tool currently under development. By demonstrating that an inexpensive software-based approach to traditional hardware-based methods is feasible and effective, while offering countless opportunities for further development and expansion, Realtime Video Underlay paves the way for accessible, flexible, and impactful visualizations.
Sports play a major role, culturally and commercially, across the globe. In the United States in particular, millions of people watch sports on broadcast, cable, or streaming platforms each year with over 74% of adults following American football and 56.6% following basketball (Lebow, 2024; Richter, 2022). The global sports industry is expected to generate $117.9 billion USD in revenue throughout 2025, with a significant portion coming from the US (Statista, 2024). The way televised sports have been presented to viewers has continuously evolved over the last 70 years. A particularly important visual feature is the addition of sports graphics overlays that update in real time, appearing to the viewer as though the visual information is being projected
Realtime Video Underlay for Accessible Television Graphics 51
onto the playing surface. One pioneer of this technology is SportsMEDIA Technology (SMT) who has been involved in the industry since 1998. SMT engineered the first realtime leaderboard graphic and later the first live scoreboard which allowed sports viewers from home to enjoy and follow game data in real time (SportsMEDIA Technology, 2024). As of 2023, SMT, and developers of similar techniques, have continued to change how sports fans enjoy and engage sports broadcasting with the addition of various elements of augmented reality. These techniques insert realtime imagery into televised athletic events helping producers and consumers track players and equipment, engage statistics and explanatory data, augment playing surface markings, and incorporate advertising into televised athletic events in novel ways.
Realtime sports graphics play a crucial role in enhancing viewer engagement, with 79% of sports fans feeling that immersive visuals increased interest in sports broadcasts, and another 57% indicating they are likely to watch content longer if virtual elements are used (Granger, 2023). Hardware-based solutions allow for impressive visuals to be created and leveraged. However, these traditional methods involve proprietary and costly equipment that limit flexibility and are largely unattainable by smaller organizations. For example, to generate and overlay the first-down line commonly seen in televised American football games a large amount of hardware is necessary including multiple powerful computers (most of them specialized), proprietary encoding devices, cameras, surface measurements, and significant wiring (HowStuffWorks, 2000; SportVision, 2020). These systems require extensive infrastructure and significant financial investments of tens of thousands of US dollars per game (Fong, Caswell; Barton, 2019) and therefore are largely used only by commercially successful sports entities and broadcasters who can afford the institutional investment in order to exploit the technological capabilities.
PROBLEM The capabilities of this technology are mostly inaccessible to other organizations such as smaller professional leagues, collegiate sports, and local school or community athletics. These barriers prevent access to technologies that could significantly enhance the production quality of smaller organizations. Whether a student athlete earns a scholarship to a university or if an athletic club can garner the community support they need can often be contingent on their ability to market themselves with professional video, television, or streaming content. Athletic professionals and stakeholders who control funding to a range of community and private sports opportunities are used to seeing professional realtime graphics and therefore may see other sports content as less valuable or less deserving for aesthetic reasons. Further, the underlying technology can be leveraged in a range of ways for helping athletes and coaches develop their team and individual skills. Professional athletes have access to this because of their commercial power, but everyone else is on the outside looking in.
52 Carlson and Beaton
This research addresses the gap between high-level and largely inaccessible sports graphics that traditionally use hardware-based methods with Realtime Video Underlay (RVU), an inexpensive software-based solution that leverages a simple camera and computer to deliver realtime sports graphics. By utilizing software-based solutions instead of systems dependent on hardware (such as the specialized computers, cameras, and wiring), RVU enables smaller organizations and institutions to elevate their production without the need for inflexible and costly hardware systems. The system’s flexibility allows for a compelling and low-cost software alternative for sports graphic broadcasting needs.
PROOF OF CONCEPT AND CASE STUDY This project was designed and executed in the Institute for New Media Studies at Fort Hays State University. Working with the university’s Tiger Media Network, the project created a software-based approach which can successfully integrate into industry-standard television and sports broadcasting tools. The NCAA Division II athletic program at Fort Hays State University does not have television contracts like most NCAA Division I programs, instead working with other universities in their athletic conference to produce live and recorded streaming of athletic events. Because of this, the more sophisticated hardware-based realtime sports graphics tools are not an option. This scenario is representative of countless smaller institutions and organizations worldwide. The first version of RVU was specifically designed for televised basketball games in order to limit the number of variables and complexity of the video feed and realtime activities. It has been completed and addresses this need. A second, more advanced version is under development which will refine the current capabilities while also adding several new ones. In particular, it will support a range of sports and realtime graphic options.
REALTIME VIDEO UNDERLAY DESIGN The philosophical approach to developing RVU is grounded in an iterative process, collaboration with partners, use of open-source and/or cost-effective tools, and low-tech replacement of complicated realtime graphics techniques wherever possible. The goal is for RVU to be compatible with a range of television and streaming platforms, easy to implement with as little technical knowledge as possible, and run on standard equipment the target audience is likely to either already own or be capable of acquiring. Partnering with Tiger Media Network helped ensure the software was tested and refined with real world use cases in mind, leading to practical solutions.
The core functionality is based on the integration of the Unity3D videogame engine and the popular OpenCV framework and libraries. Programming in Unity is done in C# but the most popular versions of OpenCV are in C++ and Python. For this reason, RVU uses OpenCV for Unity by developer Enox Software, which adequately wraps the underlying C++ libraries into the C# framework. The combination of Unity and OpenCV allow for the project to be developed for a range of hardware and operating systems. The OpenCV library is crucial for ensuring overlayed virtual effects and objects align with live footage, providing still image
Realtime Video Underlay for Accessible Television Graphics 53
and video processing in realtime to give the sense the virtual objects exist physically within a sports broadcast (Mohamad, Saman, Hitam, Telipot, 2015).
Technical Implementation Combining 3D Model and 2D Video The core process for RVU begins with creating an accurate 3D model of the basketball court. Physical measurements of the court were taken as was the location of the physical camera in the stadium (a mounted camera, that does not move, and is wired into the Tiger Media Network system, was used for the first version of this project). A virtual camera was created in Unity and then coupled to the 3D model of the court. Physical specifications of the real camera were applied to the virtual camera in order to recreate the view of the physical camera in the stadium. Simple markings were added to the virtual court to aid in alignment.
The next step involves capturing a video feed from the physical camera. To avoid reliance on expensive or proprietary equipment, RVU uses the NDI Tools software suite to capture the camera feed from the Tiger Media Network cameras in live operation, or simply the native webcam interface when directly connected to the computer. This allows RVU to use the relatively lightweight input of a physical or virtual USB webcam, which is widely supported across software and hardware combinations, regardless of what physical camera is employed (see Figure 1). Once the video feed is pulled into RVU using an OpenCV WebCamTexture, the virtual court can be moved, rotated, and scaled to align with the physical court (see Figure 2).
The video feed is then iteratively processed frame by frame using several OpenCV tools. First, the players and on-court objects are identified and tracked using background subtraction from a known still image of an empty court which has been cropped to just the relevant area for performance reasons. Experiments were done to compare the Gaussian Mixture-based Background/Foreground Segmentation algorithm (implemented as cv::BackgroundSubtractorMOG2) and K-Nearest Neighbour method (implemented as cv::BackgroundSubtractorKNN) along with best threshold values. This results in a black and white foreground mask where players appear white against a black background. This image can then be processed, as needed, with various contour tracing algorithms to help refine the mask to just players (e.g., removing the basketball by removing blobs that are too small to be a player, or using human identifying tools).
Next, all black background pixels are turned transparent to create cutouts of the players. In the current version of RVU, the player cutouts are transformed from white to green, creating an image that can be natively sent to a television broadcast system to make use of green screen tools and overlaid onto the frame (see Figure 2). When processing is internal (as will be important in the next version of RVU) this step is skipped as the transparent pixels are already enough for further processing.
Finally, one of two things happens. For external processing (using the green screen and a television broadcast system) the green layer is laid directly over the realtime video feed of the game. For internal processing (where the entire
54 Carlson and Beaton
workflow happens inside a single computer) the green screen is unnecessary and instead the transparent version of the foreground mask is used with the original incoming video source. In either case, the final step is similar. Realtime graphics (see Designs and Features section below) are drawn onto the virtual court so that they are rendered with the correct 3D perspective of being viewed from the physical camera (as if the graphics were drawn on the court). Then, the foreground mask is placed over these graphics to “cut holes” in them wherever players exist (see Figure 3). By cutting a player- shaped hole in the graphics and then aligning them over the original video footage, it appears that players are on top of the realtime graphics giving the effect that the images are actually on the court and under the people looking much the same as expensive, professional realtime sports graphics tools.
Figure 1: A still frame from a live basketball game as seen from the static physical camera.Figure 2: The live video feed is placed under the virtual court and made transparent for alignment and calibration purposes. The players are covered by green screen compatible pixels as defined in the foreground subtraction and contour processing.
Realtime Video Underlay for Accessible Television Graphics 55
Figure 3: A realtime graphic of a marker is generated by rendering it on the 3D court in unity and then cutting a hole in it using the contoured mask created in the previous step. The realtime graphics are laid over the original video feed giving the impression the player is over the marker.
DESIGNS AND FEATURES RVU contains several key features that are simple but useful (see Figure 4):
Marker Underlays: A virtual marker can be used to highlight players or key areas on a playing surface, providing broadcasters with an intuitive way to track movements and emphasize important aspects of a game. The flexibility of marker tracking ensures that it can be used across various sports and broadcasting scenarios. Rather than implementing a computationally expensive tracking system, the marker is simply placed by a user applying a finger or stylus to a touch screen. By following a player on the touch interface, a user can accurately place the marker under a player or object without any need for additional computer power.
Image Underlays: Custom graphics, such as logos or game statistics, can be displayed on the virtual court as though they physically exist, allowing broadcasters to provide valuable information to their viewers. Pulling from realtime data sources and implementing render textures allows underlays to be tailored to reflect realtime game conditions.
Text Underlays: Dynamic text underlays allow for real-time information to appear directly onto the court. Whether highlighting player statistics, game data, or strategic insights, text underlays serve as a crucial tool for broadcasters to convey essential information while enhancing viewer engagement.
Drawing Underlays: 2D and 3D drawing capabilities allow broadcasters to annotate plays and movements. This feature is particularly useful for explaining strategies and analysing key moments during a broadcast. These tools are designed to allow traditional telecaster abilities to be integrated into the realtime visuals without the need for additional tools.
56 Carlson and Beaton
These features can be expanded and changed, allowing RVU to have immense scalability and flexibility over traditional hardware-based solutions both in current and future use cases. Figure 4 showcases the interface of RVU along with each of its features in action. These functionalities are not only effective but are designed with future additions in mind.
Figure 4: The user interface and example uses of realtime video underlay showcase image and text underlay, drawing tools, and the marker. The interface is designed to be used by a human operator on a touch screen.
ANALYSIS AND FUTURE WORK Evaluation of Realtime Video Underlay demonstrates its viability as a software-based alternative to traditional hardware broadcasting solutions. Testing in collaboration with Tiger Media Network allowed for confirmation of integration with existing broadcasting workflows. Key findings show that RVU reliance on consumer-grade equipment significantly reduces costs compared to hardware-based solutions while allowing for customization and flexibility.
There are some weaknesses that still need to be addressed. Syncing realtime graphics with live video can be difficult because it demands fast processing times. This is partially mitigated with frame dropping, but it is still not always seamless sometimes introducing drift in the underlays. Contour tracing of the foreground mask can be made less accurate by dynamic lighting conditions, shadows, and unexpected activity on the court. Several approaches to enhancing this capability are being developed in the next version. Realtime graphics sometimes appear notably “fake” or “computer generated.” This is not a surprise since the goal is not to trick viewers into believing the images are real, but graphics that are too obviously different than the video feed can be distracting. This issue is complicated because it is caused by several independent factors including 3D rendering materials, quality of the original video, and aesthetic choices that must be made in any visual communication.
The next version of RVU looks to include solutions to these issues as well as new functionality: the ability to use a dynamic physical camera by integrating low-overhead augmented reality techniques appropriated from mobile device
Realtime Video Underlay for Accessible Television Graphics 57
frameworks, camera zooming through the use of realtime cropping of high resolution video sources, multiple inputs to allow for a range of underlays to be managed in realtime, and agnostic techniques that allow for a range of sports to be included.
CONCLUSION
Realtime Video Underlay aims to bridge the gap between traditionally expensive and inaccessible hardware solutions for sports broadcasting. Leveraging inexpensive software methods, Realtime Video Underlay provides a cost effective, software-based solution for smaller organizations to access advanced graphics and ways of enhancing viewer engagement. This project and associated research not only highlight the practical benefits of software-driven solutions but also demonstrates the transformative potential of integrating advanced and engaging graphic capabilities into everyday broadcasting environments. By prioritizing accessibility and innovation, it empowers organizations of all sizes to enhance their productions and connect with audiences in new and meaningful ways.
ACKNOWLEDGMENT
The authors would like to thank Tiger Media Network at Fort Hays State University for their collaboration on aspects of connecting the project to industry standard television broadcast tools: Nick Schwien, Russell Heitmann, and Trever Rohn.
REFERENCES
Fong, J., Caswell, E. Barton, G. (2019) The NFL’s virtual first-down line, explained, Vox: https://www.vox.com/2016/2/6/10919538/nfl-yellow-first-down-line-espn/. Granger, J. (June 23, 2023) New Study finds AR graphics, sports analysis, and replay
crucial for engaging younger audiences. Vizrt: https://www.vizrt.com/vizrt/press- center/engaging-younger-sports-audiences/. HowStuffWorks. (April 1, 2000) How do they superimpose the first-down line onto the field on televised football games? HowStuffWorks: https:// entertainment.howstuffworks.com/question225.htm/. Lebow, S. (November 12, 2024) More than 100 million people will watch live sports via digital this year. EMARKETER: https://www.emarketer.com/content/ 100-million-watch-live-sports-digital/. Mohamad, M., Saman, M. Y. M., Hitam, M. S. Telipot, M. (2015) A Review on
OpenCV. Terengganu: Universitas Malaysia Terengganu, 3, p. 1. Richter, F. (August 29, 2022) Which sports do Americans follow? Statista: https:// www.statista.com/chart/28107/sports-followed-by-americans/. SportsMEDIA Technology. (2024) About - SMT (SportsMEDIA Technology):
https://smt.com/about/. SportVision. (2020) 1st and ten line – how it works. SportVision: https://web.
archive.org/web/20091014141628/http://www.sportvision.com/main_frames/ products/1stand10line_works.htm [archived version from original: http://www. sportvision.com/main_frames/products/1stand10line_works.htm]. Statista. (2024). Sports – Worldwide. Statista Market Forecast: https://www.statista.
com/outlook/amo/sports/worldwide/.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors and Systems Interaction
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-68-7
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional