Why Traditional Approaches to Continuous Deployment Don’t Work Today
Editor’s note: This blog post is the first in a series about Palantir Apollo, aimed at unpacking what Apollo is, the problems it solves, and how its novel approach to software deployment can provide a competitive advantage in an increasingly complex and fragmented software landscape.
In this first post, we describe where traditional, linear approaches to continuous deployment (CD) fall short, and explain how Apollo rethinks the way software is deployed to truly solve the challenges arising from the new, complex realities of modern software development and deployment.
The traditional approach: embracing automation through “linear pipelines”
There have been amazing advances in CD over the past decade that have made it possible for changes to get from commit to production at an unprecedented speed. These advances helped software teams get new features to customers faster and more safely, while facilitating immediate feedback on how recent code changes actually work in production.
However, under the hood, most traditional CD tooling still looks largely the same as it did ten years ago: linear pipelines. It was designed assuming that software was linearly flowing through a sequential process of testing, compilation, publishing, and deployment. Over time, more parts of the deployment process were automated with policy scripts layered on top of pipelines, and better UIs were created to provide a cleaner developer experience, but at its core, the approach that traditional CD takes has not changed significantly.
The linear, “assembly line” model can only take us so far, however. The core challenge of linear pipelines today is that they fail to account for the astronomical rise in complexity with which software is both developed and where it is deployed. Specifically:
- The move away from monoliths to service-oriented architectures (SOA) means an individual update to a single service cannot be “linearly” deployed straight to a testing or production environment without first considering the implication its change has on potentially dozens of other dependencies. SOAs are valuable in that they enable development teams to ship features independently, enabling your software to evolve at the rate of the fastest movers and getting new features to customers faster. They also make your platform more modular, reusable, and resilient. However, more puzzle pieces can mean more complexity; managing large numbers of interrelated services is not trivial — and it is certainly not linear.
- It is now common practice for a single piece of software to be deployed to a number of different environments. In a growing push to meet customers where they are, it is now not uncommon for an organization to offer their software across multiple cloud providers and infrastructure types, each with varying levels of connectivity and access, as well as higher demands for compliance and data governance. This introduces a host of challenges that were not present when enterprise software sat in single on-premise data centers, or even in the dream single-cloud “SaaS” world that is quickly fizzling. The prospect of adding new, possibly heterogeneous environments as deployment targets, each with their own specific constraints/requirements, adds exponential complexity to deployment models.
The overarching consequence of these major evolutions is more complexity, and linear pipelines, notwithstanding all their benefits, have proven insufficient in handling them.
Where Traditional Methods Fall Short
Managing Service Oriented Architecture Complexity
It’s hard to safely manage deploying a platform made up of many distinct, possibly interrelated services, and the complexity scales exponentially with the number of services.
Some challenges include:
- Cross-service dependencies, both for deployment and networking. In linear CD pipeline approaches, managing cross-service dependencies between pipelines is extremely difficult. At best, you string various pipelines together in a chain, which becomes very brittle and is not scalable. Handling rollbacks without breaking API compatibility becomes an extremely difficult exercise, especially if you aim to do it in an automated manner.
- Configuration at scale, especially when managing database URIs for dependent services across multiple environments. The number of configuration points that you need to set URIs for equates to the number of edges in your graph of services multiplied by the number of environments you have. Relying on humans to configure each of these by hand can make it very brittle and cumbersome. Another pain point is managing customer-specific configuration when you have more than one instance of each service across various deployment targets.
- Lack of standardization across pipelines. Many services means you have to have many pipelines. Pipeline skew quickly emerges as each pipeline evolves independently from the others. When teams create new pipelines, they usually clone a previous pipeline and then modify it to suit the needs of the new service or team. Additionally, linear pipelines tightly couple everything together — it’s difficult to create pieces that you can standardize, reuse or substitute. This can be problematic, because there is often no common knowledge across teams in terms of how their services get deployed; there are many different paths to production. Every time an engineer joins a new team, they have to spend time figuring out how the new team’s pipelines work. Debugging pipelines also becomes more challenging. This lack of standardization across pipelines creates more complexity and overhead, costing you time and money.
- Maintaining awareness of many moving parts — What exactly is deployed where, and what is its state? Who owns what and should be contacted to resolve issues? How long should teams have to maintain backwards API compatibility?
- Asset dependencies, such as static files used for the front end.
- Managing database schema version dependencies, among other challenges.
Environment and Business Complexity
Difficult to scale at pace of business need: Services and environments are coupled within pipelines. This is problematic when your deployment model requires you to deploy to many environments. Consider when your company needs to stand up a new environment. If you have a single pipeline, you tack on each new environment to the end of it. If you have many pipelines, then you have to add each new environment to each pipeline. This is neither efficient, nor scalable, and also means that your engineers are in the critical path of new business growth.
Challenge of deploying in various configurations: If you need to deploy your platform in different configurations, with different feature sets, pipelines are a nightmare. Oftentimes, different customers may require different parts of your platform and, with a linear pipeline based approach, it is difficult to deploy different subsets of your platform in a modular manner.
Ballooning pipeline complexity: Pipelines get even more complicated when you add various compliance controls, security policies, approval steps, etc., causing pipelines to become humongous and often requiring teams to write automations scripts to manage them. Your teams are then left maintaining hundreds of pipelines and automation scripts.
The linear nature of pipelines also makes them rigid — because they’re designed from left to right, it’s complicated to have a piece running in parallel that triggers something in the middle. Achieving this often requires more scripts on top of, or around, the pipeline.
How Apollo Solves This
At Palantir, we believe that software is only becoming more complex and demands of where it deploys is only going to increase, and therefore linear pipelines are insufficient. This is why we built Apollo. It was built on several key principles and components that help your business efficiently, resiliently, and securely deploy your software wherever it needs to go:
- Separation of concerns — Decouple software and environments. Apollo models the world in terms of software packages and environments. Developers know best how their software should be upgraded and how it should behave, so they define service specifications and constraints alongside their code. Operators know best what is important to their environments and customers, so they define environment specifications and constraints. Operators choose which services should be installed in the environment and Apollo will ensure they’re installed safely. Apollo will flag if constraints are not met, and often automatically resolve them. Everyone stays within their zone of subject matter expertise, and neither persona is reliant on the other.
- Hide complexity while maintaining flexibility — Release channels. Apollo’s concept of Release Channels is a way to strike the optimal balance between stability and velocity. Release channels give you a way of modeling risk, and using more risky channels as canaries. Environments are subscribed to a given release channel, defining at what point point they will get new product releases for each of their installed services. You can also subscribe to a different release channel per product. This allows an environment owner to take more risks on specific aspects of the platform, e.g., a customer that cares about specific features being on the cutting edge but is otherwise more risk averse.
- Automate wherever possible while retaining ultimate control — Guardrails. Security and compliance are first class concepts with Apollo; it ultimately aligns incentives between developers, operators, security, compliance, and quality teams. Apollo provides several gates into production so your teams can define the policies and workflows that must be abided by. Apollo aims to automate process wherever possible so a human is not required in the loop, while maintaining all necessary security and compliance controls (e.g., contextually aware change management features).
It also provides several points for platform teams to set different types of golden paths, such as default release channels.
The next Palantir Apollo series blog post provides a technical deep dive into how Apollo rethinks this entire problem space to achieve CD for modern software realities.
Interested in Apollo for your organization? Learn more and book an Apollo demo on our website.
