Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

AutoGen-based AI assistant for improving the interactions between BIM and project teams for design coordination

1 Dept. of Architectural Engineering, The Pennsylvania State Univ., PA 16801, USA ↗
2 Dept. of Electrical and Computer Engineering, University of Houston, TX 77004, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1005750

Abstract

In the construction industry, building information modeling (BIM) has been widely utilized in design coordination. However, this process is time-consuming to query the required element information and still requires the support of the BIM coordinator. Meanwhile, during diverse participants’ discussions, it is challenging to record knowledge and experiences residing in their minds and timely respond to them in the BIM model. GPT-based Large Language Models (LLMs) enable providing automatic solutions but lack accuracy and consistency, specifically for the construction domain. To bridge these gaps, we propose to develop an AI BIM coordinator by integrating basic construction knowledge and skillset into the current AutoGen model. It aims to alleviate high-skill requirements and specific functions of traditional BIM development while enhancing the interdisciplinary interpretability and performance of AI models. Specifically, we first identify the frequent and common interactions between BIM and project teams during the design coordination meetings. Correspondingly, we build the skillset that includes basic functions regarding building element semantic, geometric, and topological information. With this skillset, our designed workflow can interpret 3D BIM space and answer specific questions from users through flexible revisions and extensions. Beyond the text responses that describe relations among elements, the BIM tool can be automatically invoked to execute this task and the model can be directly built in the 3D environment for stakeholders’ discussions in the design coordination meetings. If failed, our designed checker agent will regenerate the code until execution is succeeded. As users continually communicate with the AI BIM coordinator and provide feedback, the assistant can collect and annotate these data for fine-tuning the current model to make it more adaptive to specific construction tasks. For validation, a prototype system is developed with building design coordination meeting data. The results demonstrate that our designed workflow has better performance in execution succeeded rate (84.62%) and accuracy (76.92%) despite consuming more time (1 min 12 secs – 3 mins 1 sec) than general agent workflow.

© 2024. Published by AHFE Open Access. All rights reserved. 1829

INTRODUCTION

In the construction project, design coordination is an important task to make sure the building design meets project stakeholders’ expectations (Khorchi & Boton, 2024; Paik et al., 2022). During the design process, building information modelling (BIM) is a commonly used tool to present the 3D space of the building (Hu et al., 2020; Teo et al., 2022). It not only contains each element’s semantic information, but also represents geometric and topological relations among these elements. Typically, in the current meeting, there is still a need for BIM coordinators’ help to interact with BIM software, like query. Additionally, after analysing meeting data based on interactions, Mehrbod pointed out that it is difficult for meeting participants to find required viewpoints to show close-up details for some issues, like clash detection (Mehrbod et al., 2019). Such navigation takes up 62% of the whole meeting time while one of the most frequent tasks within navigation is changing viewpoint. It is thus time-consuming and labour tedious. Meanwhile, during the meeting, there are many conversations among participants, reflecting how different scopes of construction knowledge and experience in their minds contribute to this meeting content. Hence, recording their insights and suggestions is required for further improvement. Although such recording can be completed either by humans or by tools, it is typically used once and unable to efficiently contribute to the following other meetings or relevant projects. According to participants’ design suggestions, it still requires some time to respond to them in the BIM model. Together with these challenges that slow down the process of issue resolution, the current human interaction with BIM in building design coordination meetings still has certain room for improvement.

To bridge the aforementioned gaps, the GPT model is promising to provide automatic solutions for tasks, with the advent of Large Language Models (LLMs) (Jiang et al., 2024; Liao et al., 2024; Taiwo et al., 2024). The common limitations are lacking accuracy and consistency. Recently, Microsoft research collaborators have developed the multi-agent conversation framework named AutoGen (Li et al., 2023; Song et al., 2024). Compared with the GPT model that mainly focuses on language understanding and generation (Chung et al., 2024; Ouyang et al., 2022; Radford et al., 2018; Raffel et al., 2020), AutoGen concentrates on automatically generating code for software development (Guo et al., 2024; Li et al., 2023; Wu et al., 2023). Therefore, GPT model-based AutoGen studio integrated with BIM provides a potential solution of automatically solving the tasks within the construction domain, according to the users’ questions. However, only integration of the current AutoGen with the BIM model cannot ensure consistency and accuracy for task-solving. Therefore, we propose to design our own BIM Agent Workflow based on AutoGen, which consists of a group of agents with their own skills, like 3D building space reasoning, and task execution in BIM. This research is based on the building design coordination scenario and aims to develop AI BIM coordinators for benefiting interactions in meetings.

AutoGen-Based AI Assistant for Improving the Interactions Between BIM 1831

DEVELOPING AI BIM COORDINATORS TO SUPPORT AUTOMATIC INTERACTION This paper proposes a BIM and AutoGen-based integrated framework to support automatic interpretation and interaction with a 3D building information, as shown in Figure 1. The overview of this framework includes: 1) users ask questions in the playground based on our built BIM Agent Workflow; 2) “bim_coordinator_assistant” receives this message and coordinates a group of agents to solve the task; 3) “bim_primary_assistant” Agent provides specific steps and generate accurate python script, with its skillset regarding building element semantic, geometric, and topological information; 4) our “task_interactor_with_bim” agent will encode the provided Python script into a string and send this message to Revit; 4) Revit received this message; 5) after this message is set into python script, the commander processes and successfully executes the script in the Revit environment; 6) “bim_assistant” agent provides the result feedback about whether the execution succeeded; 7) if execution failed, the checker will regenerate the code until “bim_assistant” agent successfully executes; 8) users provide their feedback on results; 9) as users continually interact with bim_coordinator_assistant and provide feedback, these data can be collected and annotated for fine-tuning the current workflow. Regarding the framework, we enhance its performance mainly by adding the basic construction knowledge skillset and customizing the system messages of agents. For validation, it is implemented into the common interactions between BIM and project teams in design coordination meetings, including information query and model creation. This research aims to alleviate high- skill requirements and specific functions of traditional BIM development while enhancing the interdisciplinary interpretability and performance of AI models.

Figure 1: Research methodology framework.
Original PDF Figure 1:

Information Query and Reasoning of 3D Space To enhance the framework’s generalization, we use the Industrial Foundation Class (IFC)-based file, which contains building element semantic, geometric,

and topological information. However, traditional AutoGen-based multi- agents do not have good performance in interpreting this kind of file content directly. Hence, when users ask the 3D space questions, its performance is not consistent and accurate. To overcome these limitations and enhance the interdisciplinary interpretability of traditional models for 3D space, we add basic construction knowledge and functions into skills, rather than specific functions referring to traditional BIM development. It can be regarded as a trade-off between the high-skill specific functions of traditional BIM development and LLM-based task solver responses. For instance, we add the skill for instructing agents to solve relevant tasks regarding building element semantic information. The developed framework can identify more specific attributes (length and volume etc.,) and solve more complicated tasks based on several semantic constraints automatically by utilizing LLM functions, as shown in Table 1. Additionally, if we change some expressions like “fifth floor”, “5F”, or “Level 5”, it can also work well. As for geometric and topological information among building elements, this framework enables accurate identification of the relative location relationships among these elements and solves more complicated and diverse tasks based on this, as shown in Table 2.

Table 1. User’s questions and result performance about building element semantic information.

Table available in the original PDF paper.

Examples of Prompts BIM Agent Workflow General Agent Workflow Results Accuracy Time Results Different expressions

64 beams in the ‘4F’ level

Correct 22 secs

Number of beams on the Fourth Floor: 0

can you help me calculate how many beams in ‘4F’ are in the file with path: “”

64 beams on the fourth floor

Correct 33 secs

calculate how many beams in fourth floor are in the file with path: “”

64 beams on level 4

Correct 28 secs

identify how many beams in level 4 are in the file with path: “” Different elements

35 columns on the fourth floor

Correct 35 secs

Number of columns on the Fourth Floor: 0

can you help me calculate how many columns in fourth floor are in the file with path: “” Diverse elements

Correct 33 secs

64 beams and 35 columns on the fifth floor

Total number of beams and columns on the Fifth Floor: 298

can you help me calculate how many beams and columns in fifth floor are in the file with path: “”

Correct 29 secs

Diverse elements and floors

Total number of beams and columns on the Fourth Floor and Fifth Floor: 1097

can you help me calculate how many beams and columns in fourth floor and fifth floor are in the file with path: “”

Fourth Floor: Beams: 64 Columns: 35 Fifth Floor: Beams: 64 Columns: 35

(Continued)

AutoGen-Based AI Assistant for Improving the Interactions Between BIM 1833

Table 1. Continued Examples of Prompts BIM Agent Workflow General Agent Workflow Results Accuracy Time Results

Table available in the original PDF paper.

Different properties

Correct 27 secs

332 walls on the fourth floor of the specified IFC file whose length property is more than 1000 mm .

can you help me calculate the number of walls whose length in property is more than 1000 mm on the fourth floor? The file path is: “”

295 walls on the fourth floor whose length property is more than 1000 mm in the specified IFC file.

Correct 24 secs

295 walls on the fifth floor whose length property is more than 1500 mm in the specified IFC file.

can you calculate the number of walls whose length in property is more than 1500 mm on the fifth floor. The file path is:- “”

187 walls on the fifth floor whose length property is more than 1500 mm in the specified IFC file.

Correct 28 secs

Multiple constraints of semantic information

Calculate the number of columns which must meet the below constraints: 1) their length in property is no lower than 3600 mm; 2) they are on the fifth floor; 3) their volume in property is greater than 1.2. The file path is: “”

Since I cannot directly inspect the IFC file or know the exact naming conventions, you might need to provide additional information on how the fifth floor, length, and volume are identified within the IFC file.

20 columns on the fifth floor that meet the specified criteria: Their length property is no lower than 3600 mm. They are located on the fifth floor. Their volume property is greater than 1.2.

Model Creation in the 3D BIM Environment We have built our own workflow to assist traditional BIM coordinators in building design coordination meetings with multi-agents based on AutoGen. The parties as users can be the sender and “bim_coordinator_assistant” can be the receiver, including a group of our designed AI agents. For example, beyond the default setting, we customize the system message of the “bim_primary_assistant” agent to “You are an AI BIM coordinator with experienced Revit development skill. You need to generate the code required by users according to the Revit API. Use global __revit__ variable with its type of UIApplication. Current Document or current UIDocument can be obtained from __revit__. After you provided the code, the checker must be used to check the provided code. If there are some errors, must revise it. When corrected code is obtained, you must encode the executable code block into strings and then send it. Reply ‘TERMINATE’ at the end when agents ‘execution succeeded’ is achieved.” Figure 2 presents an example of the integration between AutoGen and BIM (Revit etc.,). One party in the building design coordination meeting as a user poses a simple question, our designed agents can successfully execute the result in the Revit environment. We compare the performance of BIM Agent Workflow (with checker), BIM Agent Workflow without checker, and General Agent Workflow. Three metrics are used for their evaluations, including execution succeeded

rate, accuracy, and time efficiency. Execution succeeded rate represents successful interactions between AutoGen and BIM while accuracy indicates that the result of the built model in Revit meets the user’s requirements and expectations. Table 3 indicates that our designed workflow has better performance in execution succeeded rate and accuracy but consumes more time than general agent workflow.

Table 2. User’s questions and result performance about building element topological relations.

Table available in the original PDF paper.

Examples of Prompts BIM Agent Workflow General Agent Workflow Results Accuracy Time Results Between two elements Correct 36 secs The output indicates that there is no direct spatial relationship found between Element 1704 and Element 164 in the IFC file. This means that neither of the elements is contained within the other, nor are they part of the same spatial structure.

can you help me identify the relative location relationship between element 1704 and element 164 in the file path: “”

The relative location relationship between element 164 and element 1704 in the IFC file is as follows: Left-Right Direction: Element 164 is on the Right side of element 1704. Front-Back Relation: Element 164 is Back relative to element 1704. Up-Down Relation: Element 164 is at the Top relative to element 1704. This information describes the spatial relationship between the two elements in terms of their relative positioning in three dimensions.

Among several elements

can you help me identify what elements are. located in front of element 164 in the file path: “”?

Correct 25 secs The script successfully identified the elements that are located in front of Element 164 in the IFC file. Here are the IDs of the elements: ‘0SBAWBHfv8dOjsMHp8RIKg’ ‘OSBAWBHfv8dOjsMHp8RIs’ ‘OSBAWBHfv8dOjsMHp8RiKb’ ‘OSBAWBHfv8dOjsMHp8RiGD’ These elements are positioned in front of Element 164 based on their X-coordinate values in the coordinate system of the IFC file.

Based on the analysis, the elements located in front of element 164 in the IFC file are: Element 1704: Positioned in front and below element 164. Element 1100: Positioned directly in front of element 164 at the same vertical level. Element 690: Positioned directly in front of element 164 at the same vertical level Element 1492: Positioned in front and below element 164. Element 1598: Positioned in front and below element 164. Element 1198: Positioned directly in front of element 164 at the same vertical level. Element 520: Positioned directly in front of element 164 at the same vertical level. These elements are identified as being in front of element 164 based on their relative front-back positioning.

Different expressions

Correct Correct

what are elements located in front of element 164? The file path is: “” find elements located in front of element 164 in the file path: “”

The elements located in front of element 164 are: Element 1100; Element 1704; Element 690; Element 1492; Element 1598; Element 1198; Element 520. These elements are positioned directly in front of element 164 based on their relative front-back relationship.

20 secs The script successfully identified the elements that are located in front of Element 164 in the IFC file. Here are the IDs of the elements: ‘OSBAWBHfv8dOjsMHp8RilT ‘OSBAWBHfv8dOjsMHp8Rijp’ ‘OSBAWBHfv8dOjsMHp8RiGD ‘OSBAWBHfv8dOjsMHp8RjR’ ‘OSBAWBHfv8dOjsMHp8RilW’ These elements are positioned in front of Element 164 based on their Y-coordinate values in the coordinate system of the IFC file.

(Continued)

AutoGen-Based AI Assistant for Improving the Interactions Between BIM 1835

Table 2. Continued Examples of Prompts BIM Agent Workflow General Agent Workflow Results Accuracy Time Results

Table available in the original PDF paper.

Among several elements

find elements located on the right side of element 775 in the file path: =

Correct 12 secs The script successfully identified the elements that are located on the right side of Element 775 in the IFC file. Here are the Dis of the elements: ‘3YzS2kdpz919PPXb6H_tpf 3YzS2kdpz9k9PPXb5k1B0N ‘OSBAWBHfv8dO;SMHp8RIK6 ‘OSBAWBHfv8dOjsMHp8RIKg’ ‘OSBAWBHfv8dOjsMHp8RIIs ‘OSBAWBHfv8dOjsMHp8RilT’ ‘OSBAWBHfv8dO −3M MHpRRik1’

Based on the analysis, the elements located on the right side of element 775 in the IFC file are: Element 164: Positioned to the right and slightly in front of element 775 . Element 1100: Positioned directly to the right of element 775 at the same vertical level. Element 690: Positioned directly to the right of element 775 at the same vertical level. Element 1492: Positioned to the right and slightly behind element 775 . Element 1598: Positioned to the right and slightly behind element 775 . Element 1198: Positioned directly to the right of element 775 at the same vertical level. Element 520 : Positioned directly to the right of element 775 at the same vertical level. These elements are identified as being on the right side of element 775 based on their relative left-right positioning.

Different noises

Correct Correct

676 secs

The elements located on the right side of element 775 are: Element 164; Element 1100; Element 690; Element 1492; Element 1598; Element 1198; Element 520 These elements are positioned to the right of element 775 based on their relative lef-right relationship.

waht are elements located on the right side of element 775 in the file path: −m waht are elements located on the irght side of element 775 in the file path: −n

‘OSBAWBHfv8dO isMHp8RiUp ’ ‘OSBAWBHfv8dOjsMHp8RiL5’ ‘OSBAWBHfv8dOjsMHp8RiLd’ ‘OSBAWBHfv8dOjsMHp8RiKr’ ‘OSBAWBHfv8dOjsMHp8RiKb’ ‘OSBAWBHfv8dOjsMHp8RiGD ‘3YzS2kdpz9k9PPXb6H_tpg’ These elements are positioned on the right side of Element 775 based on their X -coordinate values in the coordinate svstem of the IFC file.

Table 3. Result performance about an example of “Create a wall” in Revit.

Table available in the original PDF paper.

Examples of Prompts: Create a wall Performance BIM Agent Workflow General Agent Workflow With Checker Without Checker

Execution succeeded rate

84.62% 53.85% exitcode: 1 (execution failed) Accuracy 76.92% 30.77% You should run this script within Revit using Time efficiency 1 min 12 secs-3 mins 1 sec

54 secs -1 min 36 secs RevitPythonShell or pyRevit.

Figure 2: The left panel presents the interaction between a meeting participant and AutoGen while the right panel shows the successfully executed result in the Revit.
Original PDF Figure 2:

CONCLUSION

The current building design coordination meeting has several challenges, such as time-consuming interactions with the BIM tool, still requiring human BIM coordinators for assistance during the discussion, and difficulty in recording the knowledge and experience in the participants’ minds. To overcome these problems, this research proposes to develop AI BIM coordinators by integrating construction-specific basic knowledge and the skillset into the existing AutoGen model. By designing the BIM Agent Workflow with different roles and incorporating essential functions related to building element semantics, geometry, and topology, the proposed framework can effectively interpret 3D BIM spaces and provide accurate responses. Although the proposed prototype is based on AutoGen and Revit, the current model can also understand the IFC files to enhance the framework’s generalization in the construction domain. The built “task_interactor_with_bim” can execute the result in the Revit. Such direct visualization of results in the 3D environment beyond text responses can let participants in the meeting quickly capture issues and timely respond to the resolutions in BIM. There are two main contributions to this study. Specifically, from the construction domain, the proposed framework can alleviate high-skill requirements and specific functions of traditional BIM development, thus allowing more non-experts to efficiently interact with BIM tools. Meanwhile, it can also enhance the interdisciplinary interpretability and performance of AI models. For validation, our results demonstrate that our designed workflow has better performance in execution succeeded rate (84.62%) and accuracy (76.92%) but consumes more time (1 min 12 secs – 3 mins 1 sec) than general agent workflow.

ACKNOWLEDGMENT

This research was funded by the U.S. National Science Foundation (NSF) via Grant 2222730. The authors gratefully acknowledge NSF’s support. Any opinions, findings, recommendations, and conclusions in this paper are those of the authors, and do not necessarily reflect the views of NSF, Penn State University, and University of Houston.

REFERENCES

Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., & Brahma, S. (2024). Scaling instruction-finetuned language models. Journal of machine learning research, 25(70), 1–53. Guo, X., Huang, K., Liu, J., Fan, W., Vélez, N., Wu, Q., Wang, H., Griffiths, T. L.,

& Wang, M. (2024). Embodied llm agents learn to cooperate in organized teams. arXiv preprint arXiv:2403.12482. Hu, Y., Castro-Lacouture, D., Eastman, C. M., & Navathe, S. B. (2020). Automatic clash correction sequence optimization using a clash dependency network. Automation in Construction, 115, 103205. Jiang, G., Ma, Z., Zhang, L., & Chen, J. (2024). EPlus-LLM: A large language model-

based computing platform for automated building energy modeling. Applied Energy, 367, 123431.

AutoGen-Based AI Assistant for Improving the Interactions Between BIM 1837

Khorchi, A., & Boton, C. (2024). An OpenBIM-based 4D approach to support coordination meetings in virtual reality environments. Journal of Building Engineering, 85, 108647. Li, B., Mellou, K., Zhang, B., Pathuri, J., & Menache, I. (2023). Large language

models for supply chain optimization. arXiv preprint arXiv:2307.03875. Li, Y., Zhang, Y., & Sun, L. (2023). Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents. arXiv preprint arXiv:2310.06500. Liao, W., Lu, X., Fei, Y., Gu, Y., & Huang, Y. (2024). Generative AI design for

building structures. Automation in Construction, 157, 105187. Mehrbod, S., Staub-French, S., Mahyar, N., & Tory, M. (2019). Characterizing interactions with BIM tools and artifacts in building design coordination meetings. Automation in Construction, 98, 195–213. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C.,

Agarwal, S., Slama, K., & Ray, A. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 27730–27744. Paik, S. M., Leviakangas, P., & Choi, J. (2022). Making most of BIM in design: Analysis of the importance of design coordination. International Journal of Construction Management, 22(12), 2225–2233. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving

language understanding by generative pre-training. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140), 1–67. Song, L., Liu, J., Zhang, J., Zhang, S., Luo, A., Wang, S., Wu, Q., & Wang, C.

(2024). Adaptive In-conversation Team Building for Language Model Agents. arXiv preprint arXiv:2405.19425. Taiwo, R., Bello, I. T., Abdulai, S. F., Yussif, A.-M., Salami, B. A., Saka, A., & Zayed, T. (2024). Generative AI in the Construction Industry: A State-of-the-art Analysis. arXiv preprint arXiv:2402.09939. Teo, Y. H., Yap, J. H., An, H., Yu, S. C. M., Zhang, L., Chang, J., & Cheong, K. H.

(2022). Enhancing the MEP coordination process with BIM technology and management strategies. Sensors, 22(13), 4936. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., & Wang, C. (2023). Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155.

Cover of the proceedings volume Human Factors in Design, Engineering, and Computing

Explore the complete digital volume

Human Factors in Design, Engineering, and Computing

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-35-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →