In this episode, Edith and Paul discuss discuss how you should use the principles of Continuous Delivery to find product/market fit for your startup, tool, or service. Along the way, they highlight common reasons startups fail to find product/market fit, from undervaluing sales and marketing teams, to simply believing that if you build it, they will come. This is episode #23 in the To Be Continuous podcast series all about continuous delivery and software development.
Using feature flags to incorporate a release strategy into your development process
The Old Way
The “old way” of software releases is characterized by an explicit waterfall hand-off between teams – from Product (functional requirements) to Engineering (build and deploy). This old way did not explicitly promote release planning in the development process. The full release burden was shifted from one team to another without plans for a feedback loop or integrated release controls. Hence, it was difficult for teams to continuously deliver software, gather feedback metrics, and take full control over software rollouts and rollbacks.
With the rise of continuous delivery, teams are integrating feature release and rollout controls into the product development process. This is ushering in a new era of DevOps where teams collaborate on release strategies during the initial design phase in order to manage a release throughout the development cycle, from local, QA, staging, and to production.
How will the feature be released? Who will be getting this feature first? Will it be gradually rolled out? Who will alpha/beta test it? What if something doesn’t go right? All of these considerations (and more) are typically not discussed in initial planning meetings. By codifying a release strategy from the start, you can ensure that your software release matriculates smoothly from build to release to sunset.
Release strategy is a DevOps concept where teams integrate feature release planning into the development process. Instead of pushing a new feature to production and being done with it, developers integrate feature flags into the development lifecycle to control their releases. Broadly speaking, feature flagging is a way to control the visibility and on/off state of a particular feature by wrapping the code in a conditional. The process of flagging encourages developers to plan for the initial feature rollout, feature improvements based on user feedback, and overall feature adoption. It basically compels teams to incorporate a release strategy into the development process.
Josh Chu, Director of Engineering at Upserve, explains how his team incorporates release strategy:
“A lot of times people would say ‘let’s roll this out’ and not think about how to actually get adoption. The fact that you have a feature flag story raises visibility. Engineers start thinking about how their feature is going to get rolled out rather than just ‘push it to production, it’s done’. The product team is encouraged to take a more hands-on approach to pre-release feedback, which directly feeds into obtaining post-rollout traction.”
Strategic Rollouts Using Feature Flags
In this section, we illustrate an example of how a release strategy can be incorporated into a traditional design, build, test, and release process.
Stage 1 – Feature Design
- Design the feature’s functionality, examine the target audience and use case, and develop a timeline for implementation.
Stage 2 – Release Strategy
- Collaborate on rollout and release strategy. Should the feature flag be a front-end flag, back-end flag, or both? (Front-end flags are mainly used to control UI visibility, while back-end flags can control APIs, configurations, and even routing). What should be behind the flag? Is this a permanent or temporary flag? (ex. permanent = ‘maintenance mode’, temporary = ‘new signup form’) Who has rights to change the flag? Is the flag intended to control a percentage rollout? How will we incorporate user feedback? How will we measure adoption and traction?
Stage 3 – Build
- Develop and integrate, using the feature flag to manage the feature’s progression through multiple development environments.
Stage 4 – Test
- Test the feature in QA and Staging, using the feature flag to control rollout and user targeting.
Stage 5 – Release
- Deploy the feature as ‘off’ in production and then implement your release and rollout strategy. This could be an incremental percentage rollout, individual user targeting, or targeting groups of users.
By adopting a release strategy, Engineering, Product, and Design teams can continuously collaborate on release and rollout plans, assessing user and performance feedback along the way. Bryan Jowers, Product Manager at AppDirect, comments on the benefits of controlled rollouts, “It allows us to bring product to market faster, test, get data, and iterate… Engineering, Product, and leadership are all looking to deliver product with low risk… and we do that with tightly controlled rollouts.”
Instead of pushing a future to production and being done with it, teams should be designing release strategies that incorporate targeted/incremental rollouts with the intention of iterating based on feedback. This creates a continuous feedback loop because teams can synthesize performance metrics and transform those metrics into better, faster iterations.
Release strategy incorporates release and rollout planning into the development process. It forces teams to plan for feature rollouts and develop methods for aggregating user feedback, analyzing metrics, and assessing traction. One of the proven ways to do this, as used by teams at Upserve and AppDirect, is to use feature flags to launch, control, and measure features from development to release.
LAUNCHDARKLY HELPS YOU BUILD BETTER SOFTWARE FASTER BY HELPING MANAGE FEATURE FLAGS AT SCALE. START YOUR FREE TRIAL NOW.
In this episode, Edith and Paul are joined by Scott Raney, Partner at Redpoint Ventures. Edith, and Paul hear from Scott why continuous delivery matters in modern software development. This is episode #21 in the To Be Continuous podcast series all about continuous delivery and software development.
Feature branching allows developers to effectively collaborate around a central code base by keeping all changes to a specific feature in its own branch. With the addition of feature flags, feature branching becomes even more powerful and faster by separating feature release management from code deployment.
The Pains of Long-Lived Branching
Traditionally, new development has been done via branching. These branches typically live for months at a time, with silos of developers attempting to work in parallel. Long-lived branches often become the “I owe you” of Gitflow: “I promise to address these merge conflicts… later.” The pain points that Git was supposed to address — merge conflicts, missing dependencies, and duplicate code — gradually emerge the longer a branch lasts. We then concentrate all the pain of reworks and conflict remedies until the end, which arguably makes the pain much worse. To avoid these long-lived branching issues, branching with feature flags enables devs to effectively manage short-lived feature branches and continuously deliver better software.
To help coordinate development, engineering teams have adopted distributed version control systems (like GitHub and BitBucket). These systems allow developers to collaborate within a central codebase and use branches to make updates or develop new features. Feature branching, therefore, has become a staple of modern development cycles because they allow developers to function without encroaching on each other’s progress.
One of the shortcomings of feature branching is that feature release management is still tied to code deployments. In isolation, feature branching forces engineers to manage software delivery within the confines of their version control system. Non-technical users can’t (and shouldn’t) manage feature visibility and distribution for customers from GitHub. It is currently not possible for developers to turn features on or off in real-time, as a way to manage risk or gradually introduce a new feature.
This diagram illustrates how developers create feature branches to manage feature development, ensuring that the current build is always in a deployable state. While every company’s branching structure will be different, the basic premise is that you branch from a repository to make a new feature.
Feature Branching Using Feature Flags
This is where the introduction of feature flags makes feature branching exceptionally powerful. Feature flagging allows developers to take full control of their feature lifecycles independent of code deployments. Application features can be enabled or disabled without requiring code to be reverted or redeployed.
This process is called feature flag driven development, where continuous delivery teams manage feature rollouts separate from code deployments.
Feature flagging allows developers to take full control of their feature lifecycles without depending on code deployments. When you merge a feature branch into master (production), it is already wrapped in a feature flag. This allows you to deploy the feature “off” and then gradually roll it out to users. It also allows you to quickly “kill” the feature if it is not working well, without having to redeploy. Other benefits include:
- Better team communication
- Developers continuously collaborate on new releases, coordinate efforts, and ensure that everyone is moving forward in tandem.
- Logical deployments to prevent blocking
- Deploy code in logical chunks without resorting to long-lived branching. Even if a feature is not fully ready, it can be flagged ‘off’, but still deployed.
- Exposed dependencies
- Short-lived branching allows you to manage dependencies by removing the ones that are unnecessary.
- Faster development
- Spend less time addressing merge conflicts and refactoring old code. Spend more time delivering features that users want.
- Risk mitigation
- A feature can be flagged throughout the entire development process, from local, QA, staging, and production. This means that it can be toggled on or off in isolation, without impacting other development environments.
- Manageable code reviews
- Because you are merging more often, code reviews become less tedious and merge conflicts less onerous.
Modern DVCS combines the benefits of short-lived branches and feature flags. Tools like GitHub and Bitbucket enable continuous delivery and short-lived branching, while feature flags help make branching faster, less risky, and more manageable.
Therefore, feature flagging does not replace branching, it is complementary and makes it more powerful. Feature branching provides the flexibility to decide when and what to release, while feature flagging lets you take full control of the release itself. Together, branching and flagging help you maintain and integrate short-lived branches faster and with less risk.
Using feature flags for granular release control and risk mitigation
Blue-green deployments have long been a proven technique to mitigate risk in software releases. By adding feature flags, developers are ushering in a new era of blue-green deployments, one with unprecedented granular control over feature releases. This article discusses how to effectively integrate feature flags into your blue-green deployment process.
At its core, a blue-green deployment is a release practice that maintains two production environments called blue and green, switching between whether the blue or green environment is live. The primary benefit of this approach is to mitigate risk and control the timing of releases. The blue version might have the new version and green the old version. If something goes wrong, you can switch back to the more stable environment.
The old way of blue-green deployments was more of an all-or-none approach. Traffic was funneled in a binary fashion, all to blue or all to green. It was also focused around timing: you could determine when you wanted to switch traffic and then switch back if something went wrong.
Blue-green deployments, therefore, are a macro level of risk mitigation. You are not focusing on testing smaller features, like a new search bar, and it’s usually something controlled by your operations team, not development.
With this blue/green deployment methodology, we are placing the burden of risk mitigation on our system architecture to test different application versions. Imagine if you just wanted to test a small search bar update, would you want to go through an entire versioning process of duplicating environments and routing traffic? This method may be overly excessive for small changes, especially if you are practicing continuous delivery and releasing multiple times per week (or per day).
To get more refined release granularity, you can complement your blue/green deployment with feature flags. These flags are conditionals (if/else statements) that compartmentalize code at the ‘feature level’, meaning you can control the display of particular features instead of relying on separate application versions.
Using feature flags, you could still manage your traffic with a load balancer with the added benefit of gradually rolling out new features to your users. For example, you could switch from green to blue with the feature flag turned “off” in blue. Then, once traffic was flowing to blue, you can turn on the feature flag and gradually release the feature to 1%, 5%, 20%… of your users until you were satisfied with the performance and feedback.
Coupling feature flags with blue/green deployments could be best used with major application releases (like upgrading from version 1 to version 2 of a platform). However, for less substantial feature changes or gradual testing, you could utilize feature flags to manage feature releases within a single production environment. This will allow you to continuously release small features while still mitigating risk. It also allows you to release faster because you can practice continuous delivery while substantially reducing risk in the release phase.
DevOpsWest is run concurrently with AgileWest and had a good mix of international and domestic attendees, who wanted to get better at software. As one attendee from Florida said “I’m not working at Google or Facebook because I want to be in Florida, but I still want to be as sharp and nimble as them.” And the fact that it allowed people from London and India to visit Las Vegas seemed to be a selling point too. Continue reading “DevOps West – DevOps goes international!” »