NFDI4DataScience

Background:

The choice of image reconstruction approach, such as Iterative Model Reconstruction (IMR) and iDose4, is critical in computed tomography (CT) as distribution shifts between them can introduce systematic bias in clinical AI models, posing a barrier to efficient, scalable AI deployment in radiology. While Image-to-Image (I2I) translation offers a path toward harmonization, efficient and translation-feasible AI in radiology demands that such approaches be validated not only on image quality metrics but also on downstream clinical task performance.

Methods:

We compared three harmonization strategies: CycleGAN, Denoising Diffusion Models, and a conventional Gaussian filter, using a paired CT dataset. Performance was evaluated using task-agnostic metrics, including Structural Similarity Index Measure (SSIM), Peak Signal-to-Noise Ratio (PSNR), Frechet Inception Distance (FID), and Sliced Wasserstein Distance (SWD). Crucially, we assessed clinical utility by measuring the downstream impact on multi-organ segmentation using Mean Relative Volume Error (RVE), focusing on consistency across reconstructed modes.

Results:

Our findings reveal a significant discrepancy between visual fidelity and clinical utility. While Diffusion and CycleGAN models achieve competitive feature-level similarity, they introduce catastrophic systematic biases in downstream segmentation, particularly in adipose tissues (Body Composition Analysis, BCA), where volume errors can reach −100%, indicating a failure to preserve critical anatomical fat structures. In contrast, the computationally efficient Gaussian filter served as the most robust baseline, maintaining minimal RVE and the highest consistency between IMR and iDose4 modes.

Conclusions:

The results suggest that in the context of efficient AI for radiology imaging, where efficiency encompasses not only computational cost but also reliability of clinical outputs, task-agnostic image quality metrics are insufficient proxies for downstream clinical performance. High-parameter generative models may introduce structural losses that remain invisible to such metrics yet critically affect clinical utility. The underperformance of generative approaches observed here is likely attributable to 3D slice inconsistencies during volumetric reconstruction and suboptimal model adaptation, rather than to an inherent limitation of I2I methods, which retain significant potential for mitigating reconstruction-related bias in radiology AI. Future work developing harmonization pipelines must therefore prioritize task-specific, clinically grounded validation as the primary benchmark—and carefully consider whether computationally simpler approaches, such as conventional filtering, may already provide comparably stable harmonization for specific use cases, before resorting to high-parameter generative solutions.

Background Reproducibility of computational algorithms is a challenging but crucial requirement for medical research and an important component of trustworthy training and application of AI algorithms. Federated Learning(FL) is commonly used to enable privacy-preserving AI in medical research. One prerequisite of reproducibility is traceability. A majority of publications on traceable FL platforms leverage blockchain technology to achieve traceable FL. In healthcare settings, resource-efficient alternatives to blockchains are possible; however, their traceability features require separate design considerations.

Methods To meet the growing demand for reproducible AI, as outlined in guidelines published by governing bodies such as the European Commission, we propose a novel concept TrainTracks. TrainTracks extends the established Personal Health Train (PHT) Platform for Analytics and Distributed Machine Learning for Enterprises (PADME) tosupport reproducible and traceable federated learning in medical research. PADME already partially supports tracing the FL and changes to the analysis algorithms used in the FL projects. We extend PADME by adding privacy-preserving change tracing of the data, metadata, and computational experiment execution through integrating it with the tools specialized in distributed data management (DataLad and MetaLad). Finally, we evaluate the proposed concept against the detailed list of requirements to analyze the advantages of the TrainTracks and the need for further design improvements.

Results Evaluation of the TrainTracks concepts’ compliance with a checklist for reproducible AI showed that TrainTracks improves the original PADME platform in 15 points out of 47 points applicable to FL. The greatest improvement was in data reproducibility, improving 10 of 12 points from no support to full support for automatic information extraction. No improvements were made to method reproducibility, aside from introducing a dedicated reproducibility repository. Experiment reproducibility saw upgrades in 5 of 30 applicable points, mainly through workflow and code traceability.

Conclusions Our concept of combining FL technology with a data versioning tool provides a structured, automated workflow that traces the FL algorithm itself, delivered algorithms, and the used data. TrainTracks demonstrated high compliance with recommendations for reproducible AI experiments, methods, and data. Our work emphasizes the importance of complete FL process traceability, as all considered aspects individually contribute to the reproducibility of the medical research. Tracking dataset versions in FL is crucial for dynamic application areas, such as medical research, where new data, e.g. electronic health records, are continuously recorded and added.

Keywords Reproducibility crisis, Traceability, Trustworthy AI, Federated learning, Personal Health Train

Dataset and Study Report

The upload consists of the two following files: 

1. The study report on the Ethical, Legal, and Societal Aspects of Data Science, as manifested through a series of Interviews conducted within the framework of NFDI4DataScience Task Area 1: Community and Training. In the interviews, data science researchers and practitioners were asked to identify and analyse the most common legal and ethical challenges encountered in DS projects and how they address them. The objective was to use the collected information to create ELSA guidelines for Data Scientists. The file consists of the study guide itself and the following supplementary material:

   1. Call for participants email

   2. Call for participants email

   3. Email to the interviewees explaining the process

   4. Consent form template

   5. Anonymity Policy Clarification email

   6. Anonymisation Protocol

2. The anonymized transcripts of the interviews

Data Science (DS) and Artificial Intelligence (AI) are transforming research, industry and society at an unprecedented pace, enabling advances in areas such as healthcare, finance, e-commerce and beyond. Despite their potential, the rapid development and widespread use of DS and AI raise (novel) issues of reliability, accuracy, copyright and data protection, and bias and discrimination, among others, see for instance [1], [2], [3], [4]. It is therefore vital for data scientists to acknowledge the Ethical, Legal and Societal Aspects (ELSA) encountered in DS and AI projects, as this can promote critical thinking and reflection, thereby ensuring that data-driven systems and their underlying technologies are developed, deployed and used responsibly. In the framework of NFDI4DataScience, specifically in the Community and Training task area, we aim to develop ELSA guidelines for data scientists [5]. In order to achieve this objective, we tried to assess the landscape by conducting interviews with researchers and practitioners in the field, aiming to identify and analyse the most common ELSA challenges encountered in DS/AI projects and how to cope with them. The interviews were semi-structured interviews as this form is well suited to our purpose of collecting experiences, reflections and opinions from the participants [6]. A total of 30 were conducted between November 2022 and February 2024. The participants came mainly from academia, but the industry was also well represented. The application domains included a.o., healthcare, finance, engineering and digital humanities. In order to systematically interpret the material for manifest and latent meanings, we used qualitative content analysis [7]. Consequently, a categorisation of the material was developed to provide the basis for this interpretation [8, p. 33]. Initial categories were developed deductively, derived from the interview guide, which itself was based on existing theory and research. Subcategories were created inductively from the interviews following initial coding with the main categories. The categories reflected the key ELSA challenges faced by data scientists, including data protection, but also more specifically fairness, transparency, consent, intellectual property, and data scientists' knowledge (and also attitudes) towards ELSA challenges and how they influence their decision-making processes in the project. The results of our analysis reveal that data scientists are generally aware of ELSA issues, some more acutely than others; for example, legal issues, especially data protection, are more prominent, especially in application domains such as healthcare; bias is considered more during the data collection and less in connection to the model used or the system deployment; issues of transparency and explainability are also crucial although not prevalent. Insight was also provided regarding interdisciplinary cooperation, institutionalised ELSA support, and project documentation. Additionally, we have recorded critical assessments of the practices followed, spanning from issues with the application of laws to the responsibility and accountability of practitioners during a project life cycle. Finally, our findings emphasise the necessity of enhancing ELSA literacy and establishing and providing a strong foundational understanding of ethical and legal principles to data scientists. Developing recommendations/best practices for data scientists was regarded as a positive first step towards this goal