Richard Dreyling Iii, Regina Erlenheim, Tanel Tammet, Ingrid Pappel (2023). AI Readiness Assessment for Data-driven Public Service Projects: Change Management and Human Elements of Procurement. In: Human Factors, Business Management and Society. AHFE Open Access, vol. 97, pp. 178–187. https://doi.org/10.54941/ahfe1003894
As technology moves forward at a breakneck pace, governments are attempting to adopt technologies in their services that are complex and have unknown ramifications, like artificial intelligence (AI). Academic research and literature regarding change management and digital transformation discusses the necessity to have agreement within an organization on the technology project that the organization is considering adopting. One method of ensuring agreement within the socio-technical system is to conduct the appropriate planning prior to procurement. This also helps to avoid starting a project without having the required technological or human capabilities within the organization. Although the motives of private and public sector organizations are different when initiating projects, the human element of data-driven projects is still key to success. With the use of AI maturity models, feasibility studies and readiness assessment methods during the planning phase, public sector organizations exploring deployment of AI solutions in their public services could be able to avoid major pitfalls of projects that do not succeed. This paper’s contribution in to investigate and analyze existing methods of feasibility studies and readiness assessments through literature and document review to see how they may be applied to evaluating AI-related projects within the context of public service delivery. With the proper AI feasibility study and readiness assessment methods in place during the planning phase, public sector organizations exploring deployment of AI solutions in their public services will be able to avoid major pitfalls of piloting projects that do not succeed. This paper’s contribution in to investigate and analyze existing methods of feasibility studies and readiness assessments through academic literature and professional document review to see how they may be applied to evaluating AI-related projects within the context of public service delivery.
Governments are increasingly attempting to adopt artificial intelligence (AI) related technologies, like machine learning (ML) to aid in the provision of public-facing services. Answering questions (chatbots), pre-filled documents, and automatic translation are just few examples of existing functionalities (Mehr, 2017). However, many projects fail or do not reach the implementa- tion stage due to barriers to adoption of technologies within the organization or government (Van Noordt, 2022) (Pencheva, 2020) (Berube et al., 2021). When organizations are not able to reach consensus within the government department, or do not have the necessary talent or technical prerequisites,
AI Readiness Assessment for Data-Driven Public Service Projects 179
AI related projects can become stalled (Sun & Medaglia, 2019) (Montoya & Rivas, 2019).
While understanding AI related technologies through a socio-technical approach provides us a potential lens to understand how humans and AI systems function as a holistic system (Crowley and Lukowicz, 2019 in Nie- haus and Wiesche, 2021), the current research paper focuses primarily on the human aspects. Essentially, the type of challenges that typically cause AI projects to fail or be discontinued are human related challenges (Burgess, 2018). Implementing AI suggests high complexity, which, in turn, mean tho- rough understanding of AI readiness factors, a complex readiness assessment, and matching organizations’ current AI readiness to the aspired goal (Jöhnk, Weißert & Wyrtki, 2021).
It is argued in the current paper that change management theories provide some potential solutions in the methods that various practitioners use to build agreement within the organization facing uncertainty in the form of a change. These strategies help them to create a realistic view of the current situation while allowing for a vision of the post-change future (Kotter, 1996) (Sarayreh et al., 2013). By instituting a process that would take place in the beginning phases of planning, institutions can gain a realistic view of the socio-technical state of the organization and the project to prepare for the project in the most effective manner. One such process conducted in many fields is to do this is through readiness assessments and feasibility studies (Bruijl, 2018) (Mukherjee & Roy, 2017).
Consequently, this paper’s contribution is to investigate and analyze exi- sting methods of AI maturity models, feasibility studies, and readiness assessments through literature and document review to see how they may be applied to evaluating AI-related projects. The context of public service delivery is addressed by answering the research question, “What existing fra- meworks exist for AI maturity models, readiness assessment and feasibility analyses that could be applied to AI projects for public services?”
The paper is divided as follows. Firstly, the AI use in public services is investigated. Secondly, the human, or social element within the wider socio-technical system is addressed through the lens of organizational change management. Thirdly, existing AI maturity models and readiness assessment frameworks will be discussed and compared.
AI USE IN PUBLIC SERVICES Artificial intelligence (AI) research has existed for a long time, and it has the potential of being possibly one of the most influential technologies of our time (Niehaus and Wiesche, 2021). Even more so, in recent years, with the increase in success of statistics-based machine learning (ML) models and algorithms together with natural language processing (NLP), the amount of research on AI has increased and crossed into different domains and sectors alike (Niehaus and Wiesche, 2021) (Zhang et al., 2022). According to the latest AI Index report (Zhang et al., 2022) by Stanford University, AI has become more affordable and higher performing than ever as the costs of training machine learning systems have become lower, while training times
shorter. The capabilities of language models have become more capable in technical terms and in the context of detecting and reflecting bias and toxi- city. And while it is possible to see a clear interest in increasing investment in AI in the private sector, governmental legislation on AI is even wide-spread than just mere few years ago (Zhang et al., 2022).
However, more concretely, one of the domains where the use of AI is increasingly being used is in relation to government. Researchers provide a taxonomy of the areas in which algorithm-based technologies can be used in government as of 2018 (Engin & Treleaven, 2018). These technologies could be used in domains as diverse as, public services, supporting civil servants, national public records, national physical infrastructure, statutes and com- pliance, as well as public policy development (Engin & Treleaven, 2018). The portion of this taxonomy that this paper discusses is the use of AI – meaning “machine learning, deep-learning, and statistical modelling” in the provision of government services to help make government services more effective and efficient (Mehr, 2017) or even automating some parts of these services including citizen interaction (Engin & Trelaeven, 2018).
Van Noordt and Misuraca (2022) claim that AI has the potential of being much more impactful to citizens in comparison to other Information and Communication Technologies (ICTs). This is due to its possible implementa- tion within governmental organizations and the core learning components of the technology. However, as Margetts and Dorobantu (2019) argue, govern- ments have struggled also with much simpler technologies. Technological innovations are indispensable for the state in order to remain relevant and “to maintain its position of authority” (Margetts and Dorobantu, 2019 p. 164) in the current data-centric world. By understanding the uses of AI and implementing it in a human-centric way, governments are able to bring positive effects on services and thereby to the lives of the people who rely on them.
THE HUMAN ELEMENT AI use in government brings similar challenges to many digital transforma- tions, including the human aspects and social elements within the organi- zations. Because of this, when technology began to have an effect on the productivity of the enterprise, there has been academic literature which stu- dies the many phenomena concerning the adoption and implementation of technology and its effects. Change management and digital transformation literature study how organizations are able to adopt technologies and change, or transform.
One of the earliest foundations in psychology of what would eventually become purposed into change management literature is the three-stage model of change put forth by Kurt Lewin (Burnes, 2020). Lewin was a social psych- ologist in the twentieth century who studied the way groups operate from a social psychological perspective. He developed a three-phase model of orga- nizational change that consisted of “unfreezing, moving, and refreezing.” As Lewin was a social psychologist and paid attention to many of the interper- sonal interactions and the various forces and factors that would affect how
AI Readiness Assessment for Data-Driven Public Service Projects 181
a change progresses even though the original focus was not organizational change (Burnes, 2020). Lewin later purposed his model to organizational change specifically. The emphasis on the personal nature of change in orga- nizations is necessary because the social element has an outsized effect on the ability to complete a transition or transformation. Kotter built on Lewin’s model of organizational change by not only adding additional steps, from three to eight, but also emphasized the importance of leadership having and communicating a vision. Kotter’s model has been used in other peer reviewed literature to affect digital transformation (Auguste, 2013).
Change management literature expresses the need for groups inside orga- nizations that are considering undergoing a change to get on the same page because when changes fail, much of the time it is due to people going back to the way that they used to conduct processes before the change (Galli, 2018). After analyzing five separate models related to change management, Galli states, “No matter the model, change will only be successful if communi- cated and accepted by employees or project team members.” (Galli, 2018, p. 129). Communication and the need for leadership and the organization to understand the direction of the change as well as the intended state are key.
Change management literature explains the necessity of getting those in the organization bought-in to a change, and one thing that can help that is to understand the macro situation facing the organization. One factor found to be in enough change management models to be placed in a general model by researchers is the need for a change (Galli, 2018). The shift in the macro environment can bring the need for an organization to create change (Kotter, 1996). Two ways to judge the macro environment are readiness indices and maturity models.
Conducting a process that resembles feasibility studies or readiness asses- sments in the beginning of the conception or design of a project has the effect of giving those involved with the project a vision for the transformation that will come while also ensuring that the prerequisites for the AI related tech- nology will be met. This goes in accordance with research from the Harvard Business School on success, according to Jim Collins, one of the key aspe- cts of successful leaders of organizations is the ability to have a vision for the organization in the future, while still maintaining a realistic vision of the present (Collins, 2009).
The value of understanding the human factor in changes and transforma- tion helps to show why opportunities to get stakeholders to understand the level of the maturity for AI transformations, the readiness toward AI pro- jects, and the feasibility of specific projects can help aid the process of AI implementation in the public sector.
MATURITY MODELS, FEASIBILITY STUDIES, AND READINESS ASSESSMENTS In the academic literature, there is not much specificity as to the level at which analysis takes place. For the purpose of this paper, the different analyses will be given labels based on the layers of focus and purposes for which the analy- sis takes place. Figure 1 below shows the levels and what question each may
Figure 1: Working differences for AI maturity models, AI readiness assessments and feasibility studies.
answer. Feasibility studies and readiness assessments are different processes with varying goals that have the potential to ensure organizations are in rela- tive agreement of the goals of a system or project as well as the current state of the organization’s level of preparedness.
Maturity models judge the level of maturity of an organization in relevant categories. AI maturity models judge how well an organization is expected to be able to handle AI by examining how far along the organization is. AI readiness assessments and feasibility studies go deeper into the more specific case of a project and prerequisites for them.
Maturity models of various types have been developed by researchers to investigate the level of maturity of companies, organizations, and industries to recommend frameworks to judge the current capabilities of an entity and recommend ways to improve (Sadiq et al., 2021). The maturity models would be the primary tool to gauge the organization’s maturity level toward artifi- cial intelligence over time. The process of finding where on a maturity model a company currently exists is called a maturity assessment, which are con- ducted by a number of consultancies and technology companies that also offer similar services related to readiness assessments and feasibility studies. However, the maturity models analyzed in this paper are primarily from peer reviewed sources because there are enough academic AI maturity models for a literature review but few available sources on readiness assessments and feasibility studies for AI and almost none in public services (Sadiq et al., 2021). Readiness assessments on the other hand, are used to measure the organi- zation’s ability to adopt a new AI related technology. Again, going through the evaluation process with stakeholders allows the organization to under- stand the specifics of what they want to do for an upcoming project and analyze whether the appropriate capabilities and prerequisites are present i.e., the determination of the readiness level of the organization.
AI Readiness Assessment for Data-Driven Public Service Projects 183
Table 1. List of consultancy originated AI readiness and maturity models.
Table available in the original PDF paper.
Organization Title IBM AI Maturity Framework for Enterprise Applications (Vaish et al., 2021); Price Waterhouse Cooper (PwC)
Responsible AI - Maturing from Theory to Practice (PwC, 2021) Deloitte AI Readiness for Government: Are You Ready for AI (Van Buren, Chew, and Eggers, 2021) Accenture The Art of AI Maturity: Advancing from practice to performance (Accenture Research, 2022) Microsoft AI Maturity and Organizations: Understanding AI Maturity (Charran & Sweetman, 2018)
The feasibility study takes place at a more micro level. These are very com- mon in the medical field in which they look at the specific requirements of a project that has been proposed to determine if all of the criteria necessary for a patient study are present (Rodriguez-Ruis et al., 2019). Following this concept, in terms of AI based projects the feasibility study aims to deter- mine the viability of a single project. This requires attaining requirements and judging capabilities against them.
CURRENT FRAMEWORKS – RESULTS OF RESEARCH By assembling the similar components of AI related maturity models, readi- ness assessments, and feasibility studies, one can see commonalities between the three and suggest a method through which organizations seeking to imple- ment AI in public services can increase their organizational maturity and readiness for AI which should lead to more projects being within the realm of feasibility.
Although the argument may exist that these are not peer-reviewed pra- ctices and documents, the purpose here is to understand the state of the field as it applies to organizations adopting AI projects. The ability for these large providers to have practices that revolve around AI adoption indica- tes a significant market with continual customer feedback and cases. Other organizations also have practices revolving around AI maturity and ado- ption. For example, on azure marketplace one can purchase a set price service from various providers to conduct an AI readiness assessment or feasibility study. However, the large organizations publicly display their methodologies, allowing for analysis.
COMMON ELEMENTS The analysis of the documents from the technology companies and consul- tancies led to an understanding that even across sectors the fundamentals of AI maturity are similar. This can be seen in Table 2. The column labeled Sadiq et al. Represents the most common elements that were derived from that systematic literature review on AI maturity models.
Table 2. Comparison of the elements of readiness contained in literature and secondary documents by companies.
Table available in the original PDF paper.
Area Sadiq et al., 2021 PwC IBM Accenture Deloitte Microsoft Data X X X X X X Analytics X X X X Technology and Tools X X X X X X Intelligent Automation X X X X Governance X X X X X X People X X X X X X Organization X X X X X X Ethics X X X X X Security X X X Strategy X X X X
The documents from Table 1, which were analyzed to fill the comparison table, all deal with the factors related to the ability for organizations to adopt AI. Although they vary greatly in the respective approaches, and whether they are descriptive or proscriptive, especially in the case of the academic litera- ture, they show the sheer number of items that need to be taken into account for an organization looking to increase its maturity level or implement an AI project.
CONSIDERATIONS FOR PUBLIC SERVICES AND POTENTIAL FRAMEWORK Governments and other organizations are adopting artificial intelligence at an increasing pace (Van Noordt & Misuraca, 2022). The existence of all of these frameworks and approaches to AI maturity suggests that there is some truth to the concept presented by Jöhnk, Weißert and Wyrtki that AI is dif- ferent than most digital transformations in its “inherent complexity” (2021, p. 1). This level of complexity becomes apparent when reading the approa- ches from all of these researchers and organizations. Even though they have similar elements, they vary greatly. While be called differently, in broad stro- kes, they agree that organizations seeking to adopt AI should have methods through which they can address the presence of data, its governance, inter- nal organization, as well as having the right people and tools and technology available. Some of these are prerequisites, like data which is a key factor, the absence of which can cancel a project before it starts.
In some cases, the people who have the competencies necessary in such a cutting-edge area may not work for the government entities seeking to adopt AI. This means that it might be necessary to engage outside parties to develop a project or solution. But even if this is the case, the responsible organization would be well advised to apply one of the techniques, whether it is place- ment of the organization on a maturity model, or conducting a readiness assessment or feasibility study it is key to ensure that everyone is in accor- dance with what is expected and what the results will be. This will also help reduce the probability of the scope of the project growing beyond its original conception, by having those from all parties agree beforehand.
AI Readiness Assessment for Data-Driven Public Service Projects 185
The importance of this is reflected in almost all of the frameworks above. In the table, these lines are called “strategy,” “organization,” and “people.” However, these three lines and the table are referred to in many ways in the above documents. Terms like organizational structure, leadership buy-in, strategy and sponsorship, culture, and people are different ways of stating the importance of everyone understanding the technology in question, what changes may occur in the organization during and after the adoption process. This reflects that it is a large-scale transition to a data-driven culture. And this means change for everyone in the organization. Without the majority, or at least key stakeholders and sponsors on board, the challenges will proliferate in accordance with the complexity of the change.
AI readiness assessment that brings up uncomfortable truths might be the most important of all. For example, if leadership wants to do a project inter- nally but there are no people with the proper competencies mentioned above, a failed project can have a large cost associated with it. And in a worse case, if there is no ethical or security strategy or maturity in place, some negative ramifications may occur to the people that the organization is intending to serve.
CONCLUSION AND FUTURE WORK The list of maturity models, readiness assessments, and feasibility studies in this paper are not exhaustive. However, they serve the purpose to demon- strate the complexity of the topic of implementing AI in public services, and how it is important to come to an internal understanding through cross functional exercises and gain support prior to starting adoption.
The limitations of the study include that all of the documents have slightly different focus, and a study could benefit from conducting interviews with all of the authors or representatives from the organizations to understand deeper how they approach their interactions and engagements with customers when they are conducting evaluations, assessments and studies. Future work would include the design of a framework that is meant for use of AI in public services that goes in enough depth to be useful but also provides flexibility for the variety of situations and applications which those implementing AI in public sector organizations.
The consultancies and technology companies offering these services begin their engagement by having a workshop with parties in the organization, across functional groups, including leadership, and getting them in the same room to discuss the current level of maturity toward AI. The discussion on the business use case, and get agreement inside of the organization regarding the appropriate factors could be included in the approach. According to the change management and digital transformation literature, it might be the process itself rather than any particular methodology or framework toward AI implementation that has the positive benefit.
ACKNOWLEDGMENT
This work has been partially conducted in the project “ICT programme” which was supported by the European Union through the European Social Fund.
REFERENCES
Accenture Research, 2022. The Art of AI Maturity: Advancing from practice to performance. Accenture. Auguste, J., 2013. Applying Kotter’s 8-step process for leading change to the digital
transformation of an orthopedic surgical practice group in Toronto, Canada. J Health Med Informat, 4(3), p. 129. Bérubé, M., Giannelia, T. and Vial, G., 2021. Barriers to the Implementation of AI in Organizations: Findings from a Delphi Study. Bruijl, D. and Gerard, H. T., 2018. The relevance of Porter’s five forces in today’s
innovative and changing business environment. Available at SSRN 3192207. Burgess, A., 2018. The Executive Guide to Artificial Intelligence How to identify and implement applications for AI in your organization. by Springer Nature. Burnes, B., 2020. The origins of Lewin’s three-step model of change. The Journal of
Applied Behavioral Science, 56(1), pp. 32–59. Charran, E., and Sweetman, S., 2018. AI Maturity and organizations Understanding AI maturity. Microsoft. Collins, J., 2009. Good to Great- (Why some companies make the leap and others
don’t). Engin, Z. and Treleaven, P., 2019. Algorithmic government: Automating public services and supporting civil servants in using data science technologies. The Computer Journal, 62(3), pp. 448–460. Galli, B. J., 2018. Change management models: A comparative analysis and concerns.
IEEE Engineering Management Review, 46(3), pp. 124–132. Jöhnk, J., Weißert, M. and Wyrtki, K., 2021. Ready or not, AI comes—an interview study of organizational AI readiness factors. Business & Information Systems Engineering, 63, pp. 5–20. Kotter, J. P., 1996. Why transformation efforts fail. The Journal of Product Innova-
tion Management, 2(13), p. 170. Margetts, H. and Dorobantu, C., 2019. Rethink government with AI. Nature, 568(7751), pp. 163–165. Mehr, H., 2017. Artificial intelligence for citizen services and government. Cam-
bridge: Harvard Kennedy School. Ash Center for Democratic Governance and Innovation. Montoya, L. and Rivas, P., 2019. November. Government AI readiness meta-analysis for Latin America and The Caribbean. In 2019 IEEE International Symposium on Technology and Society (ISTAS) (pp. 1–8). IEEE. Mukherjee, M. and Roy, S., 2017. Feasibility studies and important aspect of project
management. International Journal of Advanced Engineering and Management, 2(4), pp. 98–100. Niehaus, F. and Wiesche, M., 2021. A Socio-Technical Perspective on Organizational Interaction with AI: A Literature Review. Pencheva, I., Esteve, M. and Mikhaylov, S. J., 2020. Big Data and AI–A transfor-
mational shift for government: So, what next for research?. Public Policy and Administration, 35(1), pp. 24–44. Price Waterhouse Cooper, 2021. Responsible AI – Maturing from theory to practice.
PwC. Rodriguez-Ruiz, A., Lång, K., Gubern-Merida, A., Teuwen, J., Broeders, M., Gennaro, G., Clauser, P., Helbich, T. H., Chevalier, M., Mertelmeier, T. and Wallis, M. G., 2019. Can we reduce the workload of mammographic screening by auto- matic identification of normal exams with artificial intelligence? A feasibility study. European radiology, 29, pp. 4825–4832.
AI Readiness Assessment for Data-Driven Public Service Projects 187
Sadiq, R. B., Safie, N., Abd Rahman, A. H. and Goudarzi, S., 2021. Artificial intelligence maturity model: a systematic literature review. PeerJ Computer Science, 7, p. e661. Sarayreh, B. H., Khudair, H. and Barakat, E. A., 2013. Comparative study: The
Kurt Lewin of change management. International Journal of Computer and Information Technology, 2(4), pp. 626–629. Sun, T. Q. and Medaglia, R., 2019. Mapping the challenges of Artificial Intelligence in the public sector: Evidence from public healthcare. Government Information Quarterly, 36(2), pp. 368–383. Vaish, R., Agrawal, A., Kapoor, S., and Parkin, B., 2021. AI maturity framework for
enterprise applications. IBM. Van Buren, E., Chew, B., and Eggers, W. 2021. AI readiness for government: Are you ready for AI? Deloitte. van Noordt, C. and Misuraca, G., 2022. Artificial intelligence for the public sector:
results of landscaping the use of AI in government across the European Union. Government Information Quarterly, 39(3), p. 101714. van Noordt, C., 2022, June. Conceptual challenges of researching Artificial Intelligence in public administrations. In DG. O 2022: The 23rd Annual International Conference on Digital Government Research (pp. 183–190). Zhang, D., Maslej, N., Brynjolfsson, E., Etchemendy, J., Lyons, T., Manyika, J.,
Ngo, H., Niebles, J. C., Sellitto, M., Sakhaee, E. and Shoham, Y., 2022. The AI Index 2022 Annual Report. AI Index Steering Committee. Stanford Institute for Human-Centered AI.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors, Business Management and Society
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-73-5
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional