Introducing Pilot-HDC: A Secure and Scalable Platform for Research Data Management

In the age of data-intensive science, the management of sensitive research data has become a central challenge across domains - ranging from neuroscience to clinical medicine, from computational modeling to personalized healthcare. The increasing complexity and scale of biomedical datasets, coupled with both existing and emerging regulatory demands such as the General Data Protection Regulation (GDPR) and the upcoming European Health Data Space (EHDS) Regulation, place unprecedented pressure on research infrastructures to support lawful, secure, and interoperable data workflows that also foster scientific collaboration. This is especially critical in areas such as neuroimaging, brain simulation, and digital twin modeling, where identifiable data is often essential to the research process.

At Indoc Research Europe gGmbH, we believe that research data management (RDM) infrastructures must not only uphold data protection standards but also actively support the FAIR principles as a foundation for high-quality, collaborative, and reproducible science. At the same time, RDM solutions must remain usable, scalable, and adaptable to the rapidly evolving digital and institutional landscape.

We know we are not alone in this. Many groups, consortia, and institutions are facing similar challenges, which is why we’ve decided to share our experience and solutions in a series of blog posts. In doing so, we will use our open-source platform technology Pilot-HDC as a concrete, real-world example.

What is Pilot-HDC ?

We developed Pilot-HDC precisely within this context: as a secure, extensible, and research-driven data platform technology to meet the needs of scientists working with complex, high-value, and often highly sensitive health data. Emerging from the HealthDataCloud initiative in the final phase of the Human Brain Project, our team continuous the active development and maintenance of Pilot-HDC and it´s deployment as the EBRAINS HealthDataCloud central node under the Horizon Europe-funded eBRAIN-Health project.

A Conceptual Overview

At its core, Pilot-HDC is a modular, web-accessible research data management platform that integrates streamlined data ingestion, secure storage, rich metadata support, fine-grained access control, collaborative analysis tools, and selective sharing. Each research project operates within its own isolated workspace, allowing researchers to upload data, annotate it with structured metadata, perform analyses using customizable environments - such as JupyterLab, Virtual Machines, or Business Intelligence tools - and manage access through intuitive role-based permissions. Users can interact with the platform via a modern graphical web interface or command-line tools.

Importantly, Pilot-HDC is designed with security and regulatory alignment at its core. It features distinct zones for each stage of the data lifecycle, encryption of data both in transit and at rest, and fine-grained identity and permission management - all supporting researchers in aligning with regulatory frameworks. While a formal GDPR compliance audit of the EBRAINS HealthDataCloud central node platform is still pending, Pilot-HDC is built to operate securely and to meet the technical requirements necessary for lawful processing of sensitive health data.

This conceptual architecture allows research groups to manage complex health datasets within a secure environment - without sacrificing the flexibility needed for real-world workflows.

Integrations with the EBRAINS ecosystem as the HDC central node

Developed by Indoc Research Europe gGmbH for and in exchange with EBRAINS and many European partners more, the platform builds on modern cloud-native design principles, containerization, and API-based interoperability and end-to-end integration with key EBRAINS services such as the EBRAINS Identity and Access Management (IAM), the EBRAINS Knowledge Graph, and openMINDS metadata models, or the EBRAINS centralized monitoring, alerting, and ticketing systems - among others.

Thanks to these unique features and integrations, the Pilot-HDC deployment serves as the central node of the EBRAINS HDC and is part of ongoing efforts to define a sustainable, compliant, and extensible European research infrastructure for health data, offering a practical and forward-looking foundation for modern research data management in neuroscience and beyond.

Try It Yourself

If you’d like to explore Pilot-HDC firsthand, you can! All it takes is an EBRAINS account. Once registered, you’ll be able to sign in directly to the EBRAINS HDC central node and start exploring Pilot-HDC´s features. For a step-by-step guide on how to get started, visit our user documentation here.

Expanding our Impact

Crucially, our team is not just building a platform - we are actively solving technical challenges that many research groups across Europe are facing today. Whether it’s implementing fine-grained access control, enabling real-time monitoring and alerting, managing traceable data lineage, aligning metadata with evolving standards, or developing secure encryption and providing you with data sharing pipelines to foster seamless collaboration - Pilot-HDC reflects years of hands-on expertise in software engineering, DevOps, and research infrastructure design.

Now, we’re looking to broaden how we share these insights - with practical examples, technical deep-dives, and open dialogue with the wider research community. This blog post marks the beginning of a regular series of explorations into the architecture, capabilities, and scientific use cases of the Pilot-HDC platform. Future posts will address this from two perspectives:

• From the user’s perspective: You’ll find hands-on walkthroughs, real-world scenarios, and short demo videos showcasing the latest feature additions and functionalities of Pilot-HDC.

• Under the hood: Our DevOps and platform engineers will share how we’ve addressed common infrastructure challenges on a much more technical level and how these solutions may apply to your own environment.

We believe that sharing these insights will help other teams navigate similar challenges while illustrating the real-world value, adaptability, and reliability of the platform we've built.

In our next post, we’ll follow the journey of a dataset through the Pilot-HDC platform, as it moves through the hands of different contributors. From the experimenter securely uploading raw data, to the data analyst processing and documenting it within a personalized Workspace environment, and finally to the project administrator sharing a curated version with a collaboration partner - each step reflects a real-world role and use case. We’ll showcase how Pilot-HDC supports this collaborative workflow seamlessly, with original recordings from our platform to bring the experience to life.

Concluding Thoughts

As European research communities move toward more secure, transparent, and collaborative data practices, platforms like Pilot-HDC will play an increasingly important role. By combining security, usability, and standards-based integration, we aim to support a future in which sensitive health data can be leveraged for meaningful scientific insights - without compromising privacy, compliance, or collaboration.

If you’re working with sensitive or high-value health data, facing similar infrastructure challenges, or simply curious about the potential of Pilot-HDC, we’d love to hear from you. Whether it’s feedback, a question, or a potential collaboration, don’t hesitate to reach out. We’re always excited to connect with like-minded colleagues and even more excited to share what we’ve learned!

---

Transparency Note

To refine and improve the structure, tone, and clarity of this article, I used OpenAI’s GPT-4o (ChatGPT). While the language was enhanced with the help of this tool, all ideas, perspectives, and content originate from my own experience and work. I believe that responsible use of such tools can support clear and effective scientific communication.