Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Usability Evaluation of FAIR Data Planning in the Data Stewardship Wizard

1 CTU, Faculty of Information Technology, Thákurova 9, Prague 6, 160 00, Czech Republic
2 CTU, Faculty of Electrical Engineering, Technická 2, Prague 6, 160 00, Czech Republic ↗
Publication year
Digital Object Identifier10.54941/ahfe1007063

Abstract

This study employs a Cognitive Walkthrough–oriented usability evaluation to examine how the Data Stewardship Wizard (DSW) supports the creation of FAIR-compliant data management plans. Although the FAIR principles (Findable, Accessible, Interoperable, Reusable) provide a widely accepted foundation for responsible data handling, their practical adoption often reveals cognitive and interaction-related challenges for users. Through a qualitative assessment, paired participants performed representative DSW tasks—project creation, questionnaire completion, model migration, and output generation. The walkthrough analysis highlighted several usability obstacles, including unclear navigation cues, limited progress feedback, and ambiguous system responses, all of which affected users’ orientation and comprehension of the tool’s structure.Insights from the evaluation informed targeted interface adjustments that improved clarity, workflow predictability, and collaboration transparency. These refinements reduced cognitive load and enhanced the overall intuitiveness of the system, contributing to broader acceptance among the DSW user community. The study underscores that usability evaluation is integral to the effective implementation of FAIR principles. By integrating cognitive analysis with data stewardship workflows, the work demonstrates how user-centred design can strengthen the practical application of FAIR guidelines and support more efficient and consistent data management planning.

INTRODUCTION

The FAIR Data Principles define a widely adopted framework for improving the quality, accessibility, and long-term value of research data. Findability relies on rich metadata and persistent identifiers to ensure that datasets can be reliably located by humans and machines. Accessibility requires that data can be obtained under clear and standardized conditions. Interoperability depends on shared formats, vocabularies, and ontologies that enable cross- system integration. Reusability, finally, is supported by explicit licensing and detailed provenance information that facilitate reliable secondary use of data.

FAIR Data Principles The FAIR concept was formalized by Wilkinson et al. (2016) as a set of guidelines for scientific data management that apply not only to datasets but also to algorithms, workflows, and software tools. Metadata occupy a central role in this framework: they must provide sufficient contextual and structural information and be stored in searchable registries. Persistent identifiers such as DOIs ensure stable referencing, while machine-readable representations and open protocols allow automated access and processing. Domain-specific and general-purpose repositories (e.g. GenBank, Zenodo) provide infrastructure for storing, curating, and disseminating data according to FAIR criteria. Interoperability is achieved through shared ontologies and controlled vocabularies, whereas reusability requires well-defined licences and documented provenance. These principles have been further discussed in relation to data quality, fitness for use, and assessment of FAIRness (Bishop & Hank, 2018; Jacobsen et al., 2020; Das, 2019; Anjaria, 2020; Boeckhout et al., 2018).

UI and UX Impact on FAIR Data User-centered design strongly influences the effectiveness of FAIR-oriented systems. Watson et al. (2020) show that UX methodologies enhance the accessibility and perceived value of FAIR services. Chauhan & Thalor (2023) demonstrate the impact of UI/UX-optimized data acquisition tools in agriculture, while Nakao et al. (2022) explore fairness-aware human- centered AI interfaces such as FairHIL. Upadhyay et al. (2023) illustrate how data-driven UI design contributes to usability in healthcare-related applications. Collectively, these studies underscore that FAIR compliance must be supported by interfaces that reduce cognitive load and facilitate efficient user interaction.

Review Conclusion FAIR principles, as defined by Wilkinson et al. (2016), provide a foundational framework for making scientific data discoverable, accessible, interoperable, and reusable. Their practical implementation requires not only robust metadata standards and machine-readable formats, but also usable, user- centered tools that support consistent application in real research contexts. Studies highlight the need for community-driven interpretation of FAIR (Jacobsen et al., 2020; Schultes et al., 2020) and show that effective UI/UX design enhances the usability and impact of FAIR-oriented systems across domains ranging from agriculture to healthcare and AI (Watson et al., 2020; Chauhan & Thalor, 2023; Nakao et al., 2022; Upadhyay et al., 2023). This aligns with broader findings in process-modeling usability, where the clarity of representations directly affects user comprehension (Pavlíček et al., 2017). While challenges persist in the technical and organizational aspects of FAIR implementation (Anjaria, 2020; Boeckhout et al., 2018), evidence indicates that FAIR continues to strengthen data stewardship practices, particularly in the life sciences, where structured and reusable data are essential.

MATERIAL AND METHODS Research Question

The study investigates the following research question:

  • “Can the Data Stewardship Wizard (DSW) be used effectively to support the creation of FAIR-compliant data management plans?”

The Data Stewardship Wizard Solution The Data Stewardship Wizard (DSW) is a data-management planning tool designed to support researchers, data stewards, and institutional experts (Pergl et al., 2019). It employs a dynamic, expert-curated questionnaire that adapts to user input, providing contextual guidance aligned with the FAIR principles— Findable, Accessible, Interoperable, and Reusable. Its objective is to lower the cognitive burden typically associated with data-management planning and to promote consistent, transparent, and sustainable data practices. A central feature of DSW is the automatic generation of Data Management Plans (DMPs), delivered in formats such as PDF or Word and tailored to specific institutional or funding-body requirements. The platform also includes FAIR-oriented metrics enabling users to assess and refine the quality of their plans. Its modular knowledge-model architecture allows institutions to customise domain-specific content, contributing to the broader ecosystem of interoperable FAIR tools. The DSW workflow, as described by Suchánek et al. (2019), consists of six key stages: (1) knowledge- model adaptation, (2) integration of updates using the migration tool, (3) generation of the tailored questionnaire, (4) answering project-specific questions, submitting feedback to knowledge-model authors, and (6) generating a complete DMP at any point in the process. This workflow is illustrated in Figure 1.

Figure 1: Workflow in the Data Stewardship Wizard (Suchánek et al., 2019).
Original PDF Figure 1:

Workflow in the Data Stewardship Wizard (Suchánek et al., 2019) we can be briefly described as:

  • Adaptation of the Knowledge Model: The standard Knowledge Model is tailored by a data steward to fit their specific research field and/or institution using the Editor.
  • Integration of Changes: The Migration tool is employed to integrate modifications made by other data stewards into the customized Knowledge Model.
  • Creation of the Questionnaire: Researchers generate a tailored questionnaire based on the customized Knowledge Model
  • Answering Questions: Researchers respond to relevant questions within the questionnaire related to their project.
  • Feedback Option: Each question provides an option for the researcher to submit feedback to the data steward who created the questionnaire.
  • Data Management Plan Generation: At any stage of the answering process, researchers can use the system’s templates to generate a finalized Data Management Plan (DMP).

Data Management According to Pergl et al. (2019), the core function of a data-management planning tool is to consolidate user inputs into a coherent and structured Data Management Plan (DMP). Conventional DMPs, often drafted manually by researchers, tend to be inconsistent, difficult to interpret, and occasionally incomplete. The Data Stewardship Wizard addresses these limitations by employing predominantly closed-ended questions, thereby reducing the need for extensive free-text input. Based on the structured responses provided, the system automatically generates a standardized DMP, improving both clarity and completeness.

Usability Testing Method for FAIR Data To evaluate the quality of the model used in this case study, a collaborative usability-testing approach was applied. This qualitative method, conducted with a small number of participants, focuses on observing user cognition, perception, and behavioural responses. Nielsen (1999) advocates such methods for their simplicity and efficiency, particularly in environments where users seek rapid orientation and logically organized information. Although collaborative techniques are most commonly used in business-process modelling (Forster et al., 2012), they are appropriate here because assessing the presentation of the data model within DSW essentially constitutes a process-oriented evaluation. In line with Pavlíček et al. (2017), usability was examined as a component of process-implementation quality. To strengthen internal validity, a paired interface-walkthrough method was employed, involving typical system users, developers, and ergonomics specialists (Bias, 1991). In this study, the walkthrough sessions were conducted by UX researchers together with experienced DSW users.

Data Collection Method The usability evaluation involved eight participants, working in pairs, who completed a predefined set of tasks within the DSW environment. Each pair consisted of researchers from CTU and students. The outcomes of all sessions were qualitatively analysed by a UX researcher and compared against the requirements identified by the broader DSW user community. The testing

Usability Evaluation of FAIR Data Planning in the Data Stewardship Wizard 45 protocol was based on a scenario comprising six primary tasks, conducted remotely in accordance with Gardner (2007) and Pavlíček et al. (2017). The study followed several procedural requirements: a maximum session duration of 60 minutes, mandatory use of the think-aloud technique (Nielsen, 1999), successful completion of all assigned tasks, and full recording of each session for subsequent analysis. The study follows the following rules:

  • The maximum time for studying is 60 minutes.
  • The participants should think aloud (Nielsen, 1999).
  • All tasks must be correctly finished.
  • Results will be recorded.
  • After finishing the test, the researcher performs the following interview:
  • Likes using the solution.
  • Dislikes using DSW.
  • Recommendation.

Reports will be joined together and compared with the DSW user community and their recommendations by the Data Stewardship Wizard (DSW) (DSW Ideas, n.d.).

RESULTS

The usability test examined the end-to-end process of initiating a project, completing the questionnaire, and producing model-based outputs. In addition, the analysis included a review of the FAQ and innovations originating from the user community.

List of Tasks Task 1 – Create a Project

  • Create a new project using the H2020 Project Template.
Figure 2: Project creation interface from the usability study.
Original PDF Figure 2:

Task 2 – Complete the Questionnaire

  • Specify the project name and abstract.
  • Identify and define the datasets that will be produced.
  • Complete the project template following a model example (e.g., Music Data Presentation).
  • Provide and edit multiple descriptions of the research project:
  • Correct the research project abstract after introducing an error.
  • Add an additional research project entry and move it to the first position.
  • Additional steps omitted for brevity, as the full questionnaire comprises approximately 100 items and cannot be fully reproduced here.

Task 3 – Preview the Document

  • Determine that the currently selected knowledge model is outdated.
  • Identify appropriate actions to resolve this issue.
  • Locate alternative knowledge models and extract key information:
  • Describe the Changelog model.
  • Identify the main model contributor.
  • Determine the number of contributors to the Common DSW Knowledge Model.
  • Additional subtasks follow the same pattern.

Task 4 – Migrate the Project

  • Select the most recent version of the knowledge model:
  • Explain how to recognize the latest release.
  • Identify the version number and its included changes.
  • Perform the model migration.
  • Approve all proposed changes.
  • Merge the resulting updates.

Task 5 – Generate Project Outputs

  • Export the Data Management Plan as a PDF.
  • Export the project data in JSON format.

Task 6 – Delete the Project

  • Remove the project from the DSW environment using the deletion functionality.

Cognitive Walkthrough Summary Table The following table presents an excerpt from the full cognitive-walkthrough evaluation. It is included as an illustrative template to demonstrate the structure of the analysis (Table 1).

Table 1: Used cognitive walkthrough template example part for DWS research.
Original PDF Table 1:

Were You Able To Complete The Task?

Are The Controls Understanda- Ble?

Task Order Will Users Understand How To Start The Task?

Will Users Know The Control Was The Correct One?

Was There Feedback After Completing The Task?

Find Settings Yes Yes Yes Yes After using the migration tool, it is clear to understand.

Recognize Danger Zone

After a few seconds. It is strange that it is located in settings.

Yes Yes Yes It is quite a challenging mental task to find it there because having the delete project option under Settings is very misleading.

Yes Yes Yes Yes Yes

Press Delete this project

Interview Results Positive Feedback: Participants appreciated the straightforward process of generating a Data Management Plan. They noted that, without a tool such as DSW, creating a DMP is significantly more demanding. Although the questionnaire is extensive and could benefit from a more wizard-like flow, users considered it sufficiently usable and adaptable. Most participants reported becoming comfortable with the interface within approximately 15 minutes. Negative Feedback: Negative feedback primarily concerned navigation, progress visibility, and collaboration features. Users reported difficulties caused by the absence of clear navigation elements, insufficient progress indicators, and a lack of filtering options, which collectively complicated task management and reduced transparency during workflow progression. Participants also noted shortcomings in permission granularity, limited transparency regarding user roles, and restricted collaborative functionalities. These issues negatively affected coordination, particularly in multi-user or larger project environments. Recommendations: The recommendations focused on enhancing navigation, workflow guidance, and collaborative functionality. Participants emphasized the need for clearer navigation controls, progress indicators, and filtering options, along with more explicit explanations of project phases and tagging mechanisms. They also highlighted the importance of strengthening collaboration and permission management through features such as comment deletion by project owners, read-only roles for external contributors, and increased transparency of permission settings, all of which would improve coordination and support more reliable multi-user interaction.

DISCUSSION

The results of the usability evaluation demonstrate that the Data Stewardship Wizard (DSW) offers substantial support for the creation of Data Management Plans, aligning with findings from prior studies addressing challenges in usability and data handling (Mchome et al., 2010; Johnson & Willey, 2011). Research teams participating in large-scale projects, such as Horizon 2020, inherently generate complex datasets requiring long-term stewardship. Unlike software development - where rules for sharing and reusability are relatively well establish - the management of heterogeneous scientific data remains considerably more complex. Although the FAIR principles formulated by Wilkinson et al. (2016) provide a robust conceptual foundation, their practical application is often hindered by technical and methodological barriers, particularly for teams lacking extensive data-management expertise. This challenge is well documented across multiple domains, including R&D (healthcare (Gençtürk et al., 2021), and life sciences (Boeckhout et al., 2018). Despite the usability limitations identified, DSW represents an effective tool for automating and standardizing the creation of DMPs, which remain essential for operationalizing the FAIR framework.

CONCLUSION

The Data Stewardship Wizard (DSW) has demonstrated strong potential as a platform for producing FAIR-oriented Data Management Plans. The collaborative cognitive walkthrough, supported by additional feedback from the DSW community, identified several usability shortcomings related to navigation, validation, and progress awareness. Addressing these issues- through enhancements such as improved navigation controls, customizable validation mechanisms, and persistent Knowledge Model identifiers-would further strengthen the tool’s overall usability. Enhancements to collaboration functions, including refined role management and comment handling, would also improve multi-user workflows. Additional features such as accessibility support, bulk document generation, and extended integration options (e.g., JSON-LD) would increase the scalability and reliability of the tool across diverse research contexts. By systematically addressing these usability gaps, DSW can further solidify its position as a key resource for implementing the FAIR principles and supporting efficient, transparent, and reusable data workflows.

ACKNOWLEDGEMENT

This research was supported by grant No. LM2023055 from the Ministry of Education, Youth and Sports of the Czech Republic.

REFERENCES

Anjaria, K. (2020). ‘Computational implementation and formalism of FAIR data stewardship principles.’ Data Technologies and Applications, 54, 193– 214. DOI: https://doi.org/10.1108/dta-09-2019-0164 Bias, R.G. (1991). ‘Interface-Walkthroughs: Efficient Collaborative Testing.’ IEEE

Software, 8, 94–95. DOI: https://doi.org/10.1109/52.84220 Bishop, B. & Hank, C. (2018) ‘Measuring FAIR Principles to Inform Fitness for Use.’

International Journal of Digital Curation, 13(1), pp. 65–75. Boeckhout, M., Zielhuis, G.A., & Bredenoord, A.L. (2018). ‘The FAIR guiding principles for data stewardship: fair enough?’ European Journal of Human Genetics, 26, 931–936. DOI: https://doi.org/10.1038/s41431-018- 0160-0 Chauhan, K., & Thalor, M.A. (2023). ‘UI/UX-centric design of in-the-field agricultural

data acquisition system.’ International Journal of Engineering Applied Sciences and Technology. DOI:10.33564/ijeast.2023.v08i06.010 Das, A.K. (2019). Data Stewardship for Open Science: Implementing FAIR Principles.

https://api.semanticscholar.org/CorpusID:159118266 Forster, S., Pinggera, J., & Weber, B. (2012). ‘Collaborative Business Process Modeling.’

Entwicklungsmethoden für Informationssysteme und deren Anwendung: Fachtagung. Available at: https://api.semanticscholar.org/CorpusID:3154244 (Accessed 2024). Gardner, J. (2007). ‘Remote Website Usability Testing – Benefits Over Traditional Methods.’ International Journal of Public Information Systems, 3, 63–73. Available at: https://api.semanticscholar.org/CorpusID:62675627 (Accessed 3 November 2024). Gençtürk, M., Teoman, A., Álvarez-Romero, C., Martínez-García, A., Calderón, C.L.,

Poblador-Plou, B., Löbe, M., & Sinaci, A.A. (2021). ‘End User Evaluation of the FAIR4Health Data Curation Tool.’ Studies in Health Technology and Informatics, 281, 8–12. DOI: https://doi.org/10.3233/SHTI210110

Jacobsen, A. et al. (2020) ‘FAIR Principles: Interpretations and Implementation Considerations.’ Data Intelligence, 2(1–2), pp. 10–29. Mchome, S., Sachdeva, S., & Bhalla, S. (2010). ‘A brief survey: Usability in healthcare.’

2010 International Conference on Electronics and Information Engineering, 1, V1-463–V1-467. DOI: https://doi.org/10.1109/ICEIE.2010.5559675 Nakao, Y., Strappelli, L., Stumpf, S., Naseer, A., Regoli, D., & Gamba, G.D. (2022).

‘Towards Responsible AI: A Design Space Exploration of Human- Centered Artificial Intelligence User Interfaces to Investigate Fairness.’ International Journal of Human–Computer Interaction, 39, 1762–1788. DOI: https://doi.org/10.1080/ 10447318.2022.2067936 Nielsen, J., & Landauer, T.K. (1993). ‘A mathematical model of the finding of usability problems.’ Proceedings of INTERACT '93 and CHI '93. DOI: https:// doi.org/10.1145/169059.169166 Pavlíček, J., Hronza, R., Pavlícková, P., & Jelínková, K. (2017). ‘The Business Process

Model Quality Metrics.’ EOMAS@CAiSE. DOI: https://doi.org/10.1007/978-3- 319-68185-6_10 Pergl, R. et al. Data Stewardship Wizard [Online] Available at: https://ds- wizard.

org/ (Accessed 5 November 2024). Schultes, E. et al. (2020) ‘Reusable FAIR Implementation Profiles as Accelerators of FAIR Convergence.’ In Proceedings of the International Conference on FAIR Data and Metadata, Springer, Cham, pp. 156–167. Suchánek, M. et al. (2019). Data Stewardship Wizard – Diagrams. Available at:

https://github.com/ds-wizard/dsw-diagrams (Accessed 2024). Upadhyay, A., Shekhawat, M., & Manhas, R. (2023). ‘Data-driven UX/UI design for a reproductive health tracker.’ 2023 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), 1–5. DOI: https://doi. org/10.1109/SCEECS57921.2023.10063134 Watson, C., Tvaranavicius, P., & Kaleem, R. (2020). ‘Designing services that are more

than FAIR with User eXperience (UX) techniques.’ DOI: https://doi.org/10.5194/ egusphere-egu2020-19682 Wilkinson, M.D. et al. (2016). ‘The FAIR Guiding Principles for scientific data management and stewardship.’ Scientific Data, 3, 160018. DOI: https://doi. org/10.1038/sdata.2016.18

Cover of the proceedings volume Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies

Explore the complete digital volume

Intelligent Human Systems Integration (IHSI 2026): Disruptive and Innovative Technologies

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-76-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →