Bryan Croft, Jeff Clarkson, Tai Nguyen, Seana Rothman (2025). Exploration of a Generative AI Assistant for Storyboarding and Scenario Generation. In: Human Factors in Design, Engineering, and Computing. AHFE Open Access, vol. 199, pp. 1254–1262. https://doi.org/10.54941/ahfe1006936
Generative Artificial Intelligence (GenAI) has significantly impacted not only creative industries but has found a unique place in military-based applications in storyboarding and scenario generation. GenAI has streamlined this process, making it faster, more accessible, and often more flexible to a larger audience of users. One key advantage is its ability to quickly generate a range of textual and visual concepts from user prompting. GenAI-based storyboards facilitate innovation and collaboration, amongst program managers, engineers, computer scientists, designers and the developers. Use of Large Language Models (LLMs) can be structured to support many features of storyboard creation especially from other perspectives beyond the entertainment business. It supports story drafting, shot list creation, image generation and assembly of the storyboard. This supports communication amongst the stakeholders. This research incorporates an application from initial story creation to the assembly of the storyboard with embedded module functionality found in ComfyUI which is a node-based interface and inference engine for GenAI. This is an open-source tool aid in the execution and integration of features and processes to reach desired outcomes that align with defined goals and objectives of Navy projects.
Storyboarding is common technique best known for its use in the film and media entertainment business but has existed in many forms throughout history (Lefevre, 2016). The common technique used is for an artist to create visual outlines of a story using a series of images to plan for shots in film, animation, or other end uses. The concept provides a visual roadmap for a production team. The storyboard serves as means to pre-visualize the essence of the narrative in the story. Often short clips of descriptive text can be included. This provides the outline to be used by various craftsman in the media industry to align everyone with the same vision. It acts as a visual script, a planning tool, and a communication tool. A storyboard is medium which allows for collaborative teaming in production but also to identify potential issues prior to execution of a scene in a film or animation sequence, or across the entire production. Common features of the storyboard include an image of each scene, notes on any dialog, actions to take place, placement in the
Exploration of a Generative AI Assistant for Storyboarding and Scenario Generation 1255
scene including background which defines camera angles and positioning to better visualize the concepts behind the story. The images represent key moments or shots in a film or animation. Notes are supplied for dialog of what is being said in the scene, a description of the action of what is happening in the scene and detains on how the camera is to view the scene such as a long shot, close-up, movement, angle of view, and so on. A storyboard benefits the production team to provide a plan on the visuals of the scene, to improve efficiency in modifications which can be done via the storyboard instead of during actual filming or animation creation. Additional benefits include clarification on the storyline which supports the identification of gaps, pacing of the story, and making sure the intent of the story is being told. This also support the production team by planning so the shots can be more efficient in time and budget.
Generative AI (GenAI) has recently provided the means to impact the methodology and production used in the creation and use of storyboarding especially in film making (Uddin et al., 2025). It permits a dramatic increase in speed and efficiency for those typically involved with storyboarding but also for Subject Matter Experts (SMEs) with knowledge and skills in particular areas but are not artistically oriented. The use of GenAI tools helps to facilitate rapid iteration and exploration of ideas while improving collaboration with the production team. GenAI can create entire storyboards in minutes which is dramatically faster than traditional methods. It becomes more accessible by removing the need for artistic skill and allowing other skilled members of the production team to create visual representation of their ideas. GenAI used correctly can provide cost-effective means to the process while more importantly allowing for rapid iteration on multiple versions of the storyboard to explore different visual approaches. This iterative process can allow changes on the fly. Collaboration in the use of GenAI is enhanced by providing a variety of team-based AI tools to allow sharing of content, real-time editing, feedback, and improved communication. The technique for image generation via GenAI has a significant place in the process by allowing automatic interpretation of scripts and then generate corresponding visuals. There do exist limitations and challenge with the use of GenAI in general and some of these limitations apply to storyboarding. Visual consistency must be maintained in characters, style, and composition across the storyboard. Since GenAI is a content creation tool it often creates differing content even with much of the same prompt text being the same or even re-running the content generation with the same prompt produces differing results. GenAI currently lack the ability to fully capture an artist’s intent which includes a nuanced control over composition, pose, and other content features which a professional artist provides. Along these lines the output for GenAI is unpredictable leading to nonsensical, misaligned, or inconsistent content which might require significant rework. These are the challenges that are embedded with the advantages of using GenAI tools.
RELATED WORKS The history of storyboarding (Lefevre, 2016) illustrates the impetus in understanding its utility in creating a narrative of an idea or concept which then must be created in some fashion. It also outlines the necessity and benefits to providing products especially in the media and entertainment business. Early on storyboarding when applied in areas such as layout design, keyframe animation, inbetweening, and colorization, often created substantial manual effort, demand for technical expertise, and a significant time investment in product creation (Tang et al., 2025). These challenges impeded the efficiency and scalability of the storyboarding concepts as seen in areas of comics, graphic novels and other learning tools (Uddin et al., 2025; Abram, 2008). Many of the storyboarding designs (Rasool, Molka- Danielson, & Smith, 2020) were also considered unstructured or would require techniques like learner centered design (Paracha & Yoshie, 2011), adaptive behaviours in serious games (Aronld, Fujima, Karsten, & Simeit, 2013), and transmedia-based methodology (Rasool, Molka-Danielson, & Smith, 2020) or other technics to improve upon the efficiency, scalability and even in the quality. Additional and more modern and digital-based techniques such as data driven systems (Asgari & Hurtut, 2024) or creating software with the intent to storyboard such concepts as planning (Wang et al., 2024) were also utilized. Even more recent, GenAI has entered the arena of storyboarding (Liang et al., 2024) in such areas as film making (Palmer & Palmer, 2025), Cel-animation (Tang et al., 2025), and to support artistic fusion to provide improved artistry in storyboarding (Bhuvaneshwaran, 2024). Most current works exploring the improvement in efficiency, scalability and improvement of storyboard creation have utilized GenAI techniques. This has also opened the field of interest in storyboarding to such areas such as the medical industry (Lin, Feng, & Liu, 2025; Birtwell, Morris, & Armitage, 2022) and in the defence industry as provided in this work. Storyboarding using GenAI has also specialized the generation of content by using sketches (Jo, Yuan, & Kim, 2022), creating 3D poses from 2D images (Mouaijah & Plessiet, 2021), improving prompt/query entries to improve visual quality and consistency (Kim, Eun, Oh, & Lee, 2025), as well as incorporating it into a storyboarding Virtual Reality Systems (Manuri, Sanna, & De Pace, 2023). The trend in these related works tends toward the ability to support an entire creative process and workflow into a single pipeline in the generation of storyboards that is useful for all stakeholders in the process. This is also a challenge taken on by this body of work for applying storyboarding capability to use in various projects within the Department of War (DoW).
PROPOSED METHODOLOGY The methodology for this work is based upon the following process: 1) determine the information necessary for a project to be storyboarded, 2) use Large Language Models (LLMs) to generate a story draft, 3) develop a shot list based upon previous process steps, 4) generate images via AI tools, 5) provide a structure within an application to assemble the storyboard.
Exploration of a Generative AI Assistant for Storyboarding and Scenario Generation 1257
Within the DoW there exists many different projects which cross over a large military domain. The unique feature of storyboarding is that it can be applied to any project within the DoW with support of Subject Matter Experts (SMEs) and the variety of skill individuals which are required within any given project. The concept is to get the entire team involved so that all aspects get covered in the storyboarding process. Use of UX/UI or User-Centered Design (UCD) personnel can provide the structure needed throughout the storyboarding process.
GenAI has proven itself in the creation of text, imagery and video content. It is a creative based software technology when structured correctly can provide the desires results. Much of the ability of GenAI comes from the integration of LLMs. LLMs provide the generated structure of the data necessary for the GenAI algorithms to utilize in the formulation of the content. This work utilities ComfyUI which is an open-source, node-based interface for running GenAI workflows based on Stable Diffusion. ComfyUI allows the integration of several LLMs which can be used, and which are based upon data from a variety of sources. Many different and usable LLMs exist that can be integrated into the ComfyUI structure. This work has examined many LLMs and has determined several the meet desirable outcomes for given projects and for general use. It has been determined that often a balance between the quality of the output and the size of the LLM, which relates to generative runtime, is required to meet project expectations but provide the required speed of use expected by end users. This work utilizes LLMs for both text generation like ChatGPT as well as for the generation of imagery and videos.
A shot list is a feature often used in the film industry and is essentially a detailed document created by the director or cinematographer which outlines every camera shot which is needed for the scene in the film. The content of the document typically uses a shot type, camera angle, camera movement, and a list of required equipment. In essence it is a blueprint for capturing each shot. A storyboard provides the visual representation from the shot list. Project team members support the creation of the shot list.
A necessary part of the storyboard is the visual representation at the various points of interest in the shot list. GenAI is well known for the creation/generation of imagery which can be based on text prompts and or other visual images that are representative of the desired output. ChatGPT, Midjourney and many other GenAI applications are currently known to provide such desired output. ComfyUI can provide access to such applications within its workflow structure. This work has utilized and examined several of the more extensive and well known GenAI-based image generators. Some key factor which has been examined are image/content consistency, avoidance of non-correct imagery (adding features which are not normal or natural), can generate futuristic but realistic imagery in projecting newer concepts of project, and the ability to stylize the imagery. Consistency is often a problem in GenAI and a method used call Low-Rank Adaptation (LoRA) has been applied in this work. This allows for fine-tuning large pre-trained models by modifying the model weights in Stable Diffusion and LLMs. This provides efficiency, resource savings, small-adapted files to use,
and more flexibility in swapping out different LoRA adapters for different tasks and styles and not having to retain the entire model. Most GenAI image generators have recently provided changes to avoid non-correct imagery but is still a factor to consider and better solutions are being sought in this work. The output from this work has proven the ability to generate future based imagery for use in the US Navy, however, much of the success has been to prompt manipulation and adjustments. Stylizing content has become a feature in many GenAI image generators and has proven to provide styles of interest.
ComfyUI provides the visual representation of its workflow. This helps support the necessary work to assemble a storyboard. This work has also provided an application which has been vetted by UI/UX/UCD experts to create the necessary interface and backend structure to assemble the storyboard in the desired format. This format not only provides the GenAI generation of content but the cell-based sequential element of the storyboard which include the generated imagery along with summarized text-based explanation regarding most cells within the storyboard. For this work, this is still an effort in progress but has provided example storyboards of up to 10 or 12 cells for more simple based scenarios.
The use of ComfyUI has provided the means to logically and visually piece together the various components necessary to make a storyboard generator work as outlined. ComfyUI provides a visual representation of modules which perform certain workflow features and the necessary inputs and outputs for those features. Besides the integration of GenAI techniques it provides the following capabilities within existing add-on modules or modules which can be created. These modules are now discussed. Output Settings module provides just what it states, the desired output and control of that output via specified settings. Prompts module allows for GenAI prompting entry points by the user to then to be fed to GenAI applications like ChatGPT. An Image Generation module is very similar and takes in and outputs what is expected form the embedded GenAI image generator. The Inpainting module reconstructs or fills in missing or damaged portions of a generated AI image. It supports the user in allowing to select areas an image and allow the GenAI to more closely match the surrounding area. The Area Composition module is a workflow technique which allows the user to apply different prompts to specific regions of an image to better control the final output. This is beneficial in complex scenes or in avoiding non- desired items in the scene. The Sketch to Image module takes the input of a simple drawing or sketch as a starting point and uses it to generate a more detailed and realistic AI image. This is useful in brainstorming efforts. The Face Swap module allows the desired face of a given individual to be swapped for the face provided by the GenAI image generator. The last but not final module currently in the version of this work’s storyboard application is the Style Image module. This module allows for the ability to transfer visual characteristics from a reference image. Such style characteristics include artistic style, color palette, and textures from other images.
This work has examined several technical papers that utilize GenAI or other techniques for storyboarding as illustrated in the related works
Exploration of a Generative AI Assistant for Storyboarding and Scenario Generation 1259
section. This work has also examined current commercial and open- source applications such as Storyboarder.ai, StoryboardHero, LXT Studio, Adobe Firefly, Krea.ai, Canva and others that are specifically built for the storyboarding process. This has helped to understand various GenAI and related capabilities for storyboarding. This would have DoW requirements which require it to be a stand alone or off the general internet to perform its work. ComfyUI was selected along with other LLMs and GenAI tools which allow the services to be integrated into a standalone (off general network/internet) AI-based workstation. This requirement has limited the use of several valued GenAI tools but has been able to match other storyboarding applications in its current state.
EXPECTED RESULTS The results to date for this work have proven to be successful and have gained interest from several projects which have expressed interest or have used this work’s application (currently called IMGENS-AI). This works has just begun it’s second year of funding and has made several modifications to the work effort over the first year. This work is funded by a Naval Innovation for Science and Engineering (NISE) internal Research and Development effort for NIWC Pacific. Internal marketing is currently being utilized which allows co-workers to login and supply prompts to generate Navy-based imagery under a variety project-based condition. An example is shown in Figure 1. This has provided interest across the Center which provides emphasis that the direction of the work is taking is proving to be successful. This image generation effort and contest is only part of the picture but allows for initial marketing to projects at the Center.
Figure 1: In field UX expert discussing UI/UX efforts for an application with Navy official.
A prototype application is currently functional as described. It is still in the design and development phase with expansion of capabilities. It has successfully shown how it can be used and direction it undertakes to become a more fully integrated storyboard and scenario generation tool within the domain of the US Navy and DoW.
The application for this work is still in progress and continue to evolve to become a seamless application for the generation of storyboards for DoW/Navy based applications. Below are a few images which illustrate a portion of the work that is currently functional and expected to become more functional over iterative design and development cycles. Figure 2 illustrates the current overall ComfyUI workflow modules which includes in sequential order: Output Settings, Prompt, Image Generation, Inpainting, Area Output, Sketch to Image, Face Swap, and Style Image modules.
Figure 2: Current ComfyUI visual workflow for storyboarding.
Another example from the IMGENS-AI work shows more detail on the Style Image module for the ComfyUI workflow. The nature of this application does allow for future expansion and development to improve upon current capabilities for storyboarding as well as integrate new storyboarding features as they become apparent from project requirements. The Style Image module in the application is illustrated in Figure 3.
Figure 3: Style Image module in current IMGENS-AI project.
CONCLUSION AND FUTURE WORK Current results and interest are indications that this work’s progress in the right direction. The second year of the effort will continue to improve
Exploration of a Generative AI Assistant for Storyboarding and Scenario Generation 1261
upon the existing features and new features deem necessary by projects which use the application. Future work also includes the integration of the capability into the NIWC Pacific High Performance Computing Center (HPCC) which will provide a certain capability of the application to personal across the Center. Further design and development efforts will continue as well as examination of Generative AI tools, LLMs, and new techniques in storyboarding become available.
ACKNOWLEDGMENT
The authors would like to acknowledge Naval Information Warfare Center Pacific for its Naval Innovation for Science and Engineering support to explore the concepts of GenAI-based storyboarding.
REFERENCES
Abram, S. (2008). “Storyboarding: Comics, Graphic Novels, and Engaging Learners”. Published in Multimedia & Internet @ Schools, May/June 2008, 15, 3, pp. 25–27. Arnold, S., Fujima, J., Karsten, A., & Simeit, H. (2013). “Adaptive Behaviour with User Modeling and Storyboarding in Serious Games”. 2013 International Conference on Signal-Image Technology & Internet-Based Systems. pp. 345–350. Doi: 10.1109. SITIS.2013.63. Asgari, M. & Hurtut, T. (2024). “A Design Language for Prototyping and
Storyboarding Data-Driven Stories”. Applied Sciences 2024, 1401387. Doi: 10.3390.app14041387. Bhuvaneshwaran, A., et. al., (2024). “Artistic Fusion: AI Powered Artistry for Story Boarding”. Proceedings of the International Conference on Sustainable Expert Systems (ICSES’24). Doi: 10.1109. ICSES63445.2024.10763187. Birtwell, K., Morris, R., & Armitage, C. J. (2022). “Combining Timelining
and Storyboarding to Create a Novel Method for Intervention DesIGn with StakehOlders: INDIGO”. International Journal of Qualitative Methods, 21, 1–10. Doi: 10.1177.16094069221127147. Jo, S., Yuan, Z., & Kim, S. (2022). “Interactive Storyboarding for Rapid Visual Story Generation”. 2022 IEEE International Conference on Consumer Electronics – Asia (ICCE-Asia), doi 10.1109/ICCE-ASIA7006.2022.9954679. Kim, S., Eun, J., Oh, C. & Lee, J. (2025). “Journey of Finding the Best
Query: Understanding the User Experience and AI Image Generation System”. International Journal of Human-Computer Interaction, 41–2, pp. 951–969. Doi: 10.1080.10447318.2024.2307670. Lefevre, P. (2016) “Storyboarding: a Critical History”, Historical Journal of Film, Radio & Television, December 2016, 36–4, pp. 712–713. Doi: 10.1080/ 01439685.1248068. Liang, Z., et. al. (2024). “StoryDiffusion: How to Support UX Storyboarding with
Generative-AI”. arXiv:2407.07672v1, 10 July 2024. http://arxiv.org/abs/2407. 07672. Lin, H., Feng, C., & Liu, Y. (2025). Generative AI-enhanced UX Storyboarding: Transforming Novice Designers’ Engagement and Reflective Thinking for IoMT Product Design”. International Journal of Technology and Design Education, May 2025, pp. 1–26.
Manuri, F., Sanna, A., & De Pace, F. (2023). “Storytelling in the Metaverse: from Desktop to Immersive Virtual Reality Storyboarding”. 2023 IEEE International Conference on Meteorology for eXtended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE). Doi: 10.1109. METROXRAINE58569.2023.10405763. Mouaijah, S. & Plessiet, C. (2021). “3D Character Posing for 2D Sketches for
Storyboarding”. 2021 IEEE International HCI and UX Conference in Indonesia (CHIuXiD), doi: 10.1109/CHIuXiD54398.2021.9650611. Palmer, R., & Palmer, A. (2025). “Intention is All You Need to Create Your Own Hollywood Blockbuster Movies”. Published by the IEEE Computer Society, April 2025, Editor Michael Zyda. pp. 79–82. doi: 10.1109. MC.2025.35342663. Paracha, S. & Yoshie, O. (2011). “Exploring the Role of Drama and Storyboarding
in Learner-Centered Scenario Generation”. Published in Intelligent Decision Technologies, 201l, 5, 3, pp. 237–252. Doi: 10.3222. IDT.2011.0109. Rasool, J., Molka-Danielsen, J., & Smith, C. H. (2020). “Designing Interfaces for Creative Learning Environments Using the Transreality Storyboarding Framework”. BDE 2020, May 29–31, 2020, Shanghai, China. Doi: 10.1145.3404512.3404530. Tang, Y. et. al., (2025). “Generative AI for Cel-Animation: A Survey”. ArXiv:2501.06250v4, 8 Oct 2025, https: //github.com/yunlong10/Awesome- AI4Animation. Uddin, S., et. al (2025). “Innovations and Challenges of AI in Film: A Methodological Framework for Future Exploration”. ACM Transactions on Multimedia Computing, Communications, & Application, July 2025, 21–7, pp. 1–55. Doi: 10.1145/3736725. Wang, W., et. al. (2024). “Idea Builder: Motivating Idea Generation and Planning for
Open-Ended Programming Projects through Storyboarding”. Proceedings of the 55th ACM Technical Symposium on Computer Science Education V.1 (SIGCSE 2024), March 2024, pp. 1402–1408. Doi: 10.1145.3626252.3630872.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors in Design, Engineering, and Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-75-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional