Stefano Filippi, Barbara Motyl (2024). Possible applications of large language models (LLMs) in engineering education: an overview. In: Training, Education, and Learning Sciences. AHFE Open Access, vol. 155, pp. 55–62. https://doi.org/10.54941/ahfe1005390
The research described in this paper explores the integration of Large Language Models (LLMs), particularly ChatGPT, into engineering education, focusing at the end on mechanical one. Investigating the widespread adoption of generative AI tools, the study systematically reviews their applications, usage patterns, and overall impact on knowledge acquisition in higher education. The research delves into specific applications, highlighting ChatGPT role in product design, development, and innovation. The analysis, based on a systematic review of 67 papers, categorizes findings into general and subject-specific applications, revealing notable instances in computer science and mechanical engineering. The study addresses key research questions, elucidating the diverse applications of ChatGPT, especially in supporting knowledge extension and creative idea generation. The findings underscore the transformative potential of LLMs in engineering education, emphasizing the need for careful implementation to balance AI assistance and human engagement. As ChatGPT continues to evolve, this work contributes to the ongoing discourse on effectively harnessing LLMs in educational settings, laying the groundwork for future exploration and guideline development.
The use of large language models (LLMs) is now spreading in various application and research areas. This exploratory work is concerned with investigating the use of these tools in engineering education. Specifically, it aims at clarifying the knowledge about these tools, the types of applications, and the purposes for which students use these tools within higher education courses.
The authors believe it is important to study whether the conscious use of these tools can have a positive impact on the training of future engineers and whether it can also become a tool for innovation design and development.
With the incoming of the latest versions of ChatGPT by OpenAI in 2023, generative AI tools have now become commonplace among students and teachers (Miao and Holmes, 2023; Qadir, 2023). The literature shows several examples of adoption of these tools in different educational fields concern- ing informatics, mathematics, medicine, economics, language study, etc. (Bahroun et al., 2023; Lo, 2023).
The research described in this paper investigates the different uses of tools like ChatGPT or similar in engineering education, focusing at the end on mechanical one afterward. Specifically, the authors are interested in under- standing which help may come from them to activities related to product design, development and innovation. Because of this, it was decided to conduct a systematic review of the literature.
The paper begins with a background on the main generative AI tools and how they work. It then continues with a systematic review of the main scientific publications dealing with the involvement of LLMs (in particular ChatGPT) in engineering education, to the point of specializing it towards mechanical engineering. The results of the analysis are then presented and discussed, and conclusions reported.
BACKGROUND
Generative AI Tools and LLMS Generative AI (GenAI) is an artificial intelligence technology that creates content in response to prompts within natural-language conversational inter- faces. Unlike systems that curate existing webpages, GenAI produces entirely new content, encompassing various forms of human thought representation, including natural language texts, images, videos, music, and software code. It is trained using data from webpages, social media conversations, and online media, analyzing statistical distributions of words, pixels, or other elements to identify and replicate common patterns, such as word associations.
The technologies underlying GenAI belong to the machine learning (ML) family of artificial intelligence (AI). Machine learning utilizes algorithms to enhance its performance continuously through data. A significant contrib- utor to recent AI advancements, like facial recognition, is a type of ML known as artificial neural networks (ANNs). ANNs draw inspiration from the human brain and its synaptic connections between neurons. There are various types of ANNs. Text generative AI employs a specialized type of ANN called a general-purpose transformer. Text-generating AI systems are commonly referred to as large language models (LLMs). Within this cate- gory, a specific type of LLM known as generative pre-trained transformer (GPT) plays a central role. This is the origin of the GPT acronym in the name ChatGPT. ChatGPT, specifically, is built upon GPT-3, a product of OpenAI and the third generation in their GPT series. The initial GPT model was launched in 2018, and the most recent, GPT-4, emerged in March 2023 [https://chat.openai.com/].
Nowadays, there are several language models based on transformer archi- tectures similar to ChatGPT. Some notable examples include Bard by Google [https://bard.google.com/chat], Alpaca by Stanford University’s Center for Research on Foundation Models (CRFM) [https://crfm.stanford.edu/2023/ 03/13/alpaca.html] and Elicit by Ought [https://ought.org/elicit]. These mod- els are often pre-trained on large datasets and fine-tuned for specific tasks. Each of them has its own strength and weakness and may be more suitable for certain applications or use cases.
Given the considerable interest in these tools, we thought it would have been interesting to analyze their current and potential use in engineering education, particularly in design-related activities, focusing on mechanical engineering.
Possible Applications of Large Language Models (LLMs) in Engineering Education 57
MATERIALS AND METHODS
Data Collection Inspired by two recent reviews on Generative AI and education (Bahroun et al., 2023; Lo, 2023), a review on the involvement of LLMs (limited to Chat GPT) in engineering education was considered.
The focus was to search for papers discussing the use of ChatGPT both in teaching and learning engineering and in concept design and innovation design referring to mechanical engineering design processes.
The review was conducted at the middle of January 2024. Two databases were used, Scopus and IEEE Xplore.
Using “Chat GPT AND education AND engineering” as the first search string in both DBs, 117 papers were found on Scopus and 88 on IEEE Xplore in the time range 2023-2024. Of these, those in English were then selected, 113 on Scopus and 88 on IEEE Xplore. Then, only those papers with the keyword “engineering education” were unpacked for further content anal- ysis. Sixty-four items were from Scopus and nine from IEEE Xplore; five duplicate papers were removed coming to a total of 68 papers.
In order to get a larger dataset to work on, a second search, triggered on the same sources, was conducted using the string “ChatGPT AND (conceptual design OR innovation design)”. This query returned 52 papers, considering only those written in English. All the papers came from Scopus. Then, dupli- cates were removed, reaching a total of 49 papers. The reason for this second search is that we were aware that some papers, although they considered the involvement of students in dealing with ChatGPT-related education activities, could not have “students”, “education”, etc. among their keywords.
Then, a total of 117 papers were selected for the next stage of the analysis. We read the abstracts of them in order to select all possible papers dealing with aspects related to engineering education. Finally, we got 67 papers to process further.
Research Questions At this point, the following research questions were formulated.
RQ1: Is it possible to clearly identify applications of ChatGPT to engi- neering education topics? If so, what are the main application areas?
RQ2: Strictly referring to mechanical engineering, how can ChatGPT be used in education?
RQ3: Is it possible to make clear and quantify the added value of Chat- GPT in designing and developing creative/innovative ideas in mechanical engineering education?
RESULTS AND DISCUSSION
The deeper analysis of the selected papers provided insights into the main results, organized by research question.
58 Stefano and Barbara
Research Question 1 (RQ1) Upon the initial reading of the 67 selected papers, a first classification was made between those addressing general applications of ChatGPT, encompass- ing aspects like use, ethics, and pedagogical implications, and those detailing the adoption in specific engineering application areas. This classification yielded 31 papers related to general applications and 36 papers to specific application areas. Among the latter, computer science emerged as the preva- lent area, featuring applications in information technology and software engineering and programming, totaling 18 papers. Mechanical engineering and concept and innovation design had six papers, followed by manage- ment/entrepreneurship with four papers and chemical engineering with two items. Additional applications were identified in biomedical engineering, mathematics and statistics.
Table 1 summarizes the observation for RQ1.
Table 1. Summarizing the observation for RQ1.
Table available in the original PDF paper.
Application area Papers
18
General aspects of engineering 31 Computer Science (information technology, software engineering and programming)
6
Mechanical engineering and engineering design (concept generation and product innovation) Management/entrepreneurship 4 Chemical engineering 2 Fundamental engineering subjects 2 Biomedical engineering 1 Mathematics 1 Statistics 1 Decision making 1
Therefore, in summary, the answer to RQ1 “is it possible to clearly identify applications of ChatGPT to engineering education topics? If so, what are the main application areas?” is that:
Chat GPT and LLMs in general can be used in engineering education for a variety of purposes, including basic knowledge retrieval, knowledge application, critical analysis, and knowledge extension.
The most common application areas for ChatGPT in engineering educa- tion are computer science, mechanical engineering, and management/en- trepreneurship.
Research Question 2 (RQ2) To answer RQ2, a more detailed analysis was conducted on the six publi- cations strictly related to mechanical engineering education. The code, titles and references of these papers are listed in Table 2.
Possible Applications of Large Language Models (LLMs) in Engineering Education 59
Table 2. Selected publication about mechanical engineering education.
Table available in the original PDF paper.
Code Title Reference P1 Exploring the application of ChatGPT in mechanical engineering education
(Gill, 2023)
(Puig-Ortiz et al., 2023) P2 ChatGPT for design, manufacturing, and education (Wang et al., 2023) P3 Chat generative pretrained transformer: extinction of the designer or rise of an augmented designer (Doanh et al., 2023) P4 Generative AI in the manufacturing process: theoretical considerations
(Bernabei et al., 2023)
P5 Students’ use of large language models in engineering education: a case study on technology acceptance, perceptions, efficacy, and detection chances
(Filippi, 2023) P6 Measuring the Impact of ChatGPT on Fostering Concept Generation in Innovative Product Design
In addressing RQ2, notable observations were made from the analysis of the papers P1, P5 and P6, which detailed experiences with university students. These works emphasized the direct involvement of students in use tests, where they engaged in specific tasks or participated in questionnaires, highlighting some key aspects.
To understand better the possible uses of ChatGPT, reference should be made to a classification of knowledge activities that can be performed based on Bloom’s revised taxonomy (Krathwohl, 2002) and proposed in papers P1 and P2. Both papers evaluate the performance and effectiveness of ChatGPT in supporting student learning using levels of increasing complexity. Based on these two proposals, the authors have reiterated and partially reformulated these classifications. The proposed levels are; Level 1 – Basic Knowledge; Level 2 – Knowledge Application; Level 3 – Critical Analysis; Level 4 – Knowledge Extension. Table 3 summarizes the levels and indicates examples of activities that may be undertaken and that were extracted from the six selected papers.
Table 3. Summarizing observation for RQ2.
Table available in the original PDF paper.
Level Activities Paper Codes P1, P2, P5 L1 - Basic knowledge Retrieval and understanding of basic concepts, accurate definitions
P1, P2, P5
L2 - Knowledge application
Synthesis and Evaluation; merging and summarizing information from various sources; texts production P1, P2, P3, P4 L3 - Critical analysis applying knowledge to practical scenarios and analysing outcomes; solved reasoning tasks involving simple formulas
L4 - Knowledge extension
P1, P2, P3, P4, P5, P6
application of knowledge in innovative ways to solve complex problems; optimization of design; new concept development
Considering the outcome of the analysis, ChatGPT demonstrates notable proficiency in answering questions that extend knowledge, showcasing its ability to synthesize new information, make inferences, and transfer knowl- edge to novel areas. Across diverse topics within design and manufacturing, ChatGPT consistently performs well in handling questions related to basic
60 Stefano and Barbara
knowledge and the application of knowledge. However, it encounters a notable challenge when confronted with questions requiring critical analysis, limiting its ability to break down information for a deep understanding.
Thus, the levels at which ChatGPT proves to be most effective are L1 and L2, particularly in information retrieval and text summarization. Conversely, for higher levels, issues are reported, including inconsistencies in information and potential errors, especially when support requests are not formulated accurately.
Therefore, in summary, the answer to RQ2, “strictly referring to mechan- ical engineering, how can ChatGPT be used in education?” is that:
ChatGPT can be used to support students in mechanical engineering edu- cation in a variety of ways, including answering questions, production of essays and summaries, coding, generating design concepts, and optimizing designs;
ChatGPT is most effective at answering questions that extend knowledge, while it is less effective at answering questions that require critical analysis.
Research Question 3 (RQ3) RQ3 aims to explore the effectiveness of ChatGPT as a supportive educa- tional tool for activities related to Level L4 of the previous classification, focusing on the generation of novel ideas or concepts for product develop- ment or optimization. For each of the six selected papers, efforts were made to elucidate and quantify the value added by ChatGPT in fostering creative and innovative ideas in mechanical engineering education.
Table 4 outlines key observations from the reading aimed at addressing the third research question. Specifically, the analysis identified several activities referring to L4 and linked them to applications dedicated to creativity and/or to the development of new concepts.
The most frequently examined activity was “generating design concepts based on functional or specific design requirements”, followed by “optimiz- ing design based on user-defined parameters”. Additional activities explored the potential to create simple functional models or basic graphic diagrams, generate working principles, and recognize similarities between solutions. Table 3 lists the design activities highlighted along with the references to the papers where they appear.
Table 4. Summarizing observation for RQ3.
Table available in the original PDF paper.
Design activities Paper Codes Generate design concepts based on functional or specific design requirements P1, P2, P3, P4, P6 Optimize design based on user-defined parameters P4, P6 Create simple graphical schemas using textual characters P1 Create function models P3 Generate working principle for a function or a subfunction P3 Recognition of similar working structures P3
Possible Applications of Large Language Models (LLMs) in Engineering Education 61
The analysis reveals that ChatGPT serves as a valuable tool for offering innovative ideas and suggestions to support the initial stages of design- ing new products and optimizing existing ones. While ChatGPT creativity may not surpass that of expert engineers, it proves beneficial for students and individuals with less experience. Furthermore, a noteworthy finding is the recognition that to harness the full potential of these tools, users need not only a sound understanding of their functionality but also strong crit- ical and analytical skills. These skills are crucial for effectively articulating problems and critically evaluating the solutions obtained. Paper P5 did not contribute to speculation about RQ3 (it does not appear in Table 4). This is because, although the research involved students thanks to a question- naire on engineering education, it does not specifically deal with creativity or innovation.
Therefore, in summary, the answer to RQ3, “is it possible to make clear and quantify the added value of ChatGPT in designing and developing creative/innovative ideas in mechanical engineering education?” is that:
ChatGPT can be used to generate creative and innovative ideas in mechan- ical engineering education, supporting the first phases of product design and development.
The most common applications for ChatGPT in this area are generat- ing design concepts, optimizing designs, and creating simple graphical schemas starting from specific and well-defined requirements.
CONCLUSION
The research described in this paper aimed to comprehensively review the different applications of generative AI tools in engineering education focusing on both engineering in general, mechanical engineering in particular, and the design field.
This study highlights the potential of Large Language Models (LLMs) as valuable support tools for students in engineering education. However, effec- tive implementation necessitates careful consideration, training, and critical analysis of the outputs of the tools involved. Striking a balance between AI assistance and human involvement is crucial, with an emphasis on foster- ing critical thinking and problem-solving skills. This work should contribute to the ongoing discourse on integrating LLMs into education and lays the groundwork for further exploration and guideline development to harness their full potential in educational settings. Delving into specifics, Chat- GPT shows promise in enhancing various activities such as human-machine collaboration, knowledge management, design innovation, and engineer- ing education within the manufacturing sector. While excelling in certain areas, such as extending knowledge and handling basic questions, challenges arise, particularly in critical analysis. These findings underscore ChatGPT transformative potential across different facets of the manufacturing indus- try, including collaboration, knowledge utilization, design innovation, and education.
62 Stefano and Barbara
In future, there is an expectation to expand the mapping of uses, especially in areas related to specific industry training, lifelong learning activities, or teacher training.
REFERENCES
Bahroun, Z., Anane, C., Ahmed, V. and Zacca, A. (2023). Transforming education: A comprehensive review of generative artificial intelligence in educational settings through bibliometric and content analysis. Sustainability, 15 (17), 12983. https: //doi.org/10.3390/su151712983 Bernabei, M., Colabianchi, S., Falegnami, A., Costantino, F. (2023). Students’ use of
large language models in engineering education: A case study on technology accep- tance, perceptions, efficacy, and detection chances. Computers and Education: Artificial Intelligence, 5, 100172. https://doi.org/10.1016/j.caeai.2023.100172 Doanh, D. C., Dufek, Z., Ejdys, J., Gineviˇcius, R., Korzynski, P., Mazurek, G., Paliszkiewicz, J., Wach, K. and Ziemba, E., (2023). Generative AI in the manufacturing process: Theoretical considerations. Engineering Management in Production and Services, 15(4), 76–89. https://doi.org/10.2478/emj-2023-0029 Filippi, S. (2023). Measuring the impact of ChatGPT on fostering concept generation
in innovative product design. Electronics, 12(16), 3535. https://doi.org/10.3390/ electronics12163535 Gill, A. S. (2023). Chat Generative Pretrained Transformer: Extinction of the Designer or Rise of an Augmented Designer. In International Design Engineering Technical Conferences and Computers and Information in Engineering Confer- ence (Vol. 87318, p. V03BT03A063). American Society of Mechanical Engineers. https://doi.org/10.1115/DETC2023-116971 Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into
practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2 Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsc i13040410 Miao, F., Holmes, W. (2023). Guidance for generative AI in education and research.
UNESCO Publishing. Puig-Ortiz, J., Pàmies-Vilà, R., Jordi Nebot, L. (2023). Exploring The Application Of Chatgpt In Mechanical Engineering Education. https://doi.org/10.21427/PP 64-KG63 Qadir, J. (2023). Engineering education in the era of ChatGPT: Promise and pitfalls of
generative AI for education. In 2023 IEEE Global Engineering Education Confer- ence (EDUCON) (pp. 1–9). IEEE. https://doi.org/10.1109/EDUCON54358.2023. 10125121 Wang, X., Anwer, N., Dai, Y., Liu, A. (2023). ChatGPT for design, manufacturing, and education. Procedia CIRP, 119, 7–14. https://doi.org/10.1016/j.procir.2023. 04.001
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Training, Education, and Learning Sciences
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-31-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional