Announcing Girder 5

A Modern and Scalable Data Management Platform

We are delighted to announce the release of Girder 5, a major rewrite that brings modern development practices, improved scalability, and a more maintainable codebase to our popular data management platform.  Girder has been the core infrastructure behind HistomicsTK and the Digital Slide Archive and is used with DIVE and other Kitware platforms.

Girder 5 represents a significant departure from previous versions, with a focus on:

• 12-factor application principles for modern cloud deployment
• Improved scalability for production environments
• Modern JavaScript tooling for the web client
• Simplified plugin architecture for faster builds and better composability
• Better developer experience for easier testing and modularization

12-Factor App Compliance for Modern Deployments

The most significant change in Girder 5 is its complete adherence to 12-factor app principles, making it far more portable and easier to scale across different environments.  This is the modern standard for scalable web applications.

• Configuration entirely via environment variables – No more config files in a myriad of locations.  Before, Girder could get its configuration from a range of sources, which made some local development easier but made it much harder to track why a particular setting had a specific value. By only using environment variables, it is much easier to replicate Girder’s state across deployments.
• Standardized logging – All logs output to stdout, letting other infrastructure handle rotation and aggregation.  In the Digital Slide Archive reference deployment, this enabled us to refactor how logs are stored and rotated, improving log visibility for the database and other services.
• Stateless processes – The core application is truly stateless, improving scalability and reliability.  Girder could always conceptually have multiple servers using a common database and asset stores, but it required sticky sessions in load balancing to prevent splitting a single user’s requests across servers.  By adopting stateless principles, Girder scales better.

Production-Ready Scalability

Girder has long used CherryPy as its internal web router.  This is a Python package that has been around for over 20 years and, when we started Girder, was one of the most widely supported frameworks.  Some parts have stood the test of time; others have not.  Specifically, besides routing web requests, CherryPy can also serve them. Under heavy load, it would return HTTP 503 errors.  

Girder 5 fixes this by replacing CherryPy with WSGI for requests and ASGI for asynchronous notifications.  Our initial field reports clearly show that we no longer get 503 errors under load and have a lighter-weight notification stream for better updates in the user interface, such as job statuses.

Modern Web Client Development

We’ve changed the way web clients are built.  Girder has always supported plugins.  Some of these are small, such as customizing the home page.  Others are substantial, such as the HistomicsUI for annotating pathology images and running ML jobs. 

In Girder 3, changing any plugin required rebuilding all plugins using a tool called Webpack.  This was slow and, for developers, tedious to write tests that checked the user interface.

Plugins are now built independently.  Developers can use whatever build tools they want; we’ve switched to Vite, which is vastly faster than Webpack.  For example, Girder 3 took 4 or 5 minutes on a modern computer to build a common set of plugins, while Girder 5 needs to be built less often and takes under 15 seconds to do the same — a 20x speedup that unlocks developer productivity.

Migration Path

The migration guide provides detailed instructions for upgrading.  If you are using HistomicsTK, you can just switch to the newer docker-compose.yaml and restart.  If you wrote your own plugins for Girder that are pure Python, you might not need to do anything.  For client plugins, follow the guide. And of course, you can reach out to us if you need specific support for a complex setup.

Conclusion

Girder 5 is more than an incremental update.  It is a major refactoring to embrace modern practices like 12-factor app design, proper WSGI compliance, and a modular build system.  The platform is easier to use, faster, and more reliable, giving you better tools to get to the results you want.

Visit our discussion forum to connect with other users, ask questions, and share your experiences, and reach out to us if you need more specific advice or help