02 Aug 2016

Feature Flag-Driven Releases

Feature flag strategy meeting

Using feature flags to incorporate a release strategy into your development process

 

The Old Way

The “old way” of software releases is characterized by an explicit waterfall hand-off between teams – from Product (functional requirements) to Engineering (build and deploy). This old way did not explicitly promote release planning in the development process. The full release burden was shifted from one team to another without plans for a feedback loop or integrated release controls.  Hence, it was difficult for teams to continuously deliver software, gather feedback metrics, and take full control over software rollouts and rollbacks.

With the rise of continuous delivery, teams are integrating feature release and rollout controls into the product development process.  This is ushering in a new era of DevOps where teams collaborate on release strategies during the initial design phase in order to manage a release throughout the development cycle, from local, QA, staging, and to production.

How will the feature be released?  Who will be getting this feature first?  Will it be gradually rolled out? Who will alpha/beta test it?  What if something doesn’t go right?  All of these considerations (and more) are typically not discussed in initial planning meetings. By codifying a release strategy from the start, you can ensure that your software release matriculates smoothly from build to release to sunset.

Release Strategy

Release strategy is a DevOps concept where teams integrate feature release planning into the development process.  Instead of pushing a new feature to production and being done with it, developers integrate feature flags into the development lifecycle to control their releases.  Broadly speaking, feature flagging is a way to control the visibility and on/off state of a particular feature by wrapping the code in a conditional.  The process of flagging encourages developers to plan for the initial feature rollout, feature improvements based on user feedback, and overall feature adoption.  It basically compels teams to incorporate a release strategy into the development process.

Josh Chu, Director of Engineering at Upserve, explains how his team incorporates release strategy:

“A lot of times people would say ‘let’s roll this out’ and not think about how to actually get adoption. The fact that you have a feature flag story raises visibility. Engineers start thinking about how their feature is going to get rolled out rather than just ‘push it to production, it’s done’. The product team is encouraged to take a more hands-on approach to pre-release feedback, which directly feeds into obtaining post-rollout traction.”

 

LaunchDarkly Rollout and Release Strategy Feature Flags and Feature Toggles

Strategic Rollouts Using Feature Flags

In this section, we illustrate an example of how a release strategy can be incorporated into a traditional design, build, test, and release process.

Stage 1 – Feature Design

  • Design the feature’s functionality, examine the target audience and use case, and develop a timeline for implementation.

Stage 2 – Release Strategy

  • Collaborate on rollout and release strategy.  Should the feature flag be a front-end flag, back-end flag, or both? (Front-end flags are mainly used to control UI visibility, while back-end flags can control APIs, configurations, and even routing).  What should be behind the flag?  Is this a permanent or temporary flag? (ex. permanent = ‘maintenance mode’, temporary = ‘new signup form’) Who has rights to change the flag?  Is the flag intended to control a percentage rollout? How will we incorporate user feedback? How will we measure adoption and traction?

Stage 3 – Build

  • Develop and integrate, using the feature flag to manage the feature’s progression through multiple development environments.

Stage 4 – Test

  • Test the feature in QA and Staging, using the feature flag to control rollout and user targeting.

Stage 5 – Release

  • Deploy the feature as ‘off’ in production and then implement your release and rollout strategy.  This could be an incremental percentage rollout, individual user targeting, or targeting groups of users.

LaunchDarkly Rollout and Release Strategy Using a Feedback Loop Feature Flags and Feature Toggles

Feedback Loop

By adopting a release strategy, Engineering, Product, and Design teams can continuously collaborate on release and rollout plans, assessing user and performance feedback along the way.  Bryan Jowers, Product Manager at AppDirect, comments on the benefits of controlled rollouts, “It allows us to bring product to market faster, test, get data, and iterate… Engineering, Product, and leadership are all looking to deliver product with low risk… and we do that with tightly controlled rollouts.”

Instead of pushing a future to production and being done with it, teams should be designing release strategies that incorporate targeted/incremental rollouts with the intention of iterating based on feedback.  This creates a continuous feedback loop because teams can synthesize performance metrics and transform those metrics into better, faster iterations.

Take-Aways

Release strategy incorporates release and rollout planning into the development process.  It forces teams to plan for feature rollouts and develop methods for aggregating user feedback, analyzing metrics, and assessing traction.  One of the proven ways to do this, as used by teams at Upserve and AppDirect, is to use feature flags to launch, control, and measure features from development to release.

 

LAUNCHDARKLY HELPS YOU BUILD BETTER SOFTWARE FASTER BY HELPING MANAGE FEATURE FLAGS AT SCALE. START YOUR FREE TRIAL NOW.

 

26 Jun 2016

Staging Servers are Dead! Long Live a Staging Server

Earlier this year, I wrote about why staging servers should die – that they actually increase risk and time and decrease quality. I’ve been very pleased with the thoughtful comments and feedback I’ve gotten about why effective continuous delivery and DevOps means no staging server. The one that made me the happiest was “Dreams exist to become reality. Here is one I’d like to achieve at work.”

Here I’ll address the feedback I received, and what’s standing in the way of achieving this dream.

 

First, there was a large agreement that “waterfall deployment” with a staging server added time and risk to the launch process.

 

Here are the questions and concerns about the practicality and validity of the “#noStaging” Dream.

 

“Don’t we need to test major infrastructure changes in Staging?”

I’m not advocating pushing untested code willy nilly into production. I am advocating for through and complete automation, continuous integration and feature flagging, with the goal to get as quickly as possible to production. The reason to kill staging servers was said very well by Joseph Ruscio, Librato CTO & co-founder  “You think of everything in advance of how a user will use a feature, and you still miss half of them.” The purpose of DevOps is to move as fast as possible from code on a developers box to customer.

 

One great case study is Librato. Librato uses feature flags to wrap features, deploying their code and then ramping volume up and down, controlling risk. The allure of a staging server is that all systems can be thoroughly tested there, “guaranteeing” a painless final deployment. However, the moment where you push code to the real world is the scariest, because the real world NEVER matches staging. By using feature flagging, Librato could de-risk a new infrastructure project. They used “Branch-by-abstraction” to ramp on and off a new infrastructure. The old way would have been to push the entire new infrastructure at a slow time like 3 am, then when something went wrong, scramble to fix, precisely when memories and tempers are short due to sleep deprivation and key people might be asleep. With feature flags, Librato could ramp up volume in the real world, with real data. Ruscio says of their rollout with feature flags, “We never got paged at night for an issue with the new system, as we were only introducing it during the day when we were available. So all issues could be addressed while we were available. With feature flags, we could even dial back risk during lunch hour.”

 

“But we need a staging server for contractual reasons”

Having a staging server for contractual reasons is arguing for an artificial artifact. As Dave Farley, co-author of Continuous Delivery” says “it’s not done until it’s delivered to users”. Lamont Lucas, FastRobot co-founder, says “media and advertising companies want final approval before a feature goes live, generally from non-technical approvers. Flagging all users coming from a gateway lets you fulfill the contractual obligation (of showing a feature to them for approval) while preventing you from having an entire legacy/pointless staging setup.”

 

“What about performance testing?”

Sean Byrnes, CEO/Founder of Outlier and the Founder of Flurry, often had to “test that new features work in a production-like environment and don’t compromise the integrity of our systems”, as Flurry had millions of users worldwide.  However, Byrnes continued, “You don’t have to load test EVERY feature to failure”.

The failure of most features is lack of load as there’s no customer interest. However for features where you do really need to know the failure rate, an alternate server other than production can be spun up ephemerally for that reason. Just don’t call it “the staging server” – it’s a production-like load system spun up for a specific reason – to test load – and then shut down after its purpose is complete.

 

“Feature flags are only good for shallow UE changes, not for Microservices”.

Actually, microservices make using staging servers even more painful as many versions of different microservices might be running across staging and production. Deger says,  “In a microservices architecture with lot of independent deployments going on, using a single staging environment would inadvertently test integrations with services in different versions, than what is currently in production. This gives a false sense of security. So we decided to not have a staging environment at all and find different ways to release features with confidence while only integrating into production.

We are constantly learning new techniques to deploy into production without a staging environment. There are many different ways of integrations that we need to care of, but overall it makes us faster and the testing and release experience is better.””

 

“Harbaugh is biased”

I advise that instead of a confusing, time consuming, expensive, redundant and ultimately unsuccessful staging server; development teams should use feature flags to move features to production and do their feature validation as quickly as possible on production. I’m not unbiased – I’m CEO of a feature flagging management platform, LaunchDarkly. However, I founded LaunchDarkly the same reason I publish articles, podcasts, and give talks on DevOps: because I care deeply about the power of feature flagging to create better quality software and reduce risk. You don’t need LaunchDarkly to feature flag – it’s easy to get started with a simple config file or open source library. Deger, is the contributor to an open source framework for feature flagging and huge proponent of no staging servers.

 

“Let’s talk when you want to do serious software development”.

Just as waterfall development once seemed the only way to guarantee success, so did waterfall deployment until recently. The most innovative and fast moving companies like Netflix, Google, and Amazon have found that they can move faster if they’re more agile. And serious? I think of software development as fun, as exciting, as liberating, transforming – with software people are connected worldwide, interacting in incredible ways, and their lives are better. I hope software never becomes too serious for me.

14 Jun 2016

DevOps West – DevOps goes international!

DevOpsWest

DevOpsWest is run concurrently with AgileWest and had a good mix of international and domestic attendees, who wanted to get better at software. As one attendee from Florida said “I’m not working at Google or Facebook because I want to be in Florida, but I still want to be as sharp and nimble as them.” And the fact that it allowed people from London and India to visit Las Vegas seemed to be a selling point too. Continue reading “DevOps West – DevOps goes international!” »

02 Jun 2016

Powering Continuous Delivery With Feature Flags

Continuous Delivery LaunchDarkly

Separating feature rollout from code deployment to mitigate risk in continuous delivery

We are in the era of continuous delivery, where we are expected to quickly deliver software that is stable and performant.  We see development teams embracing a suite of continuous integration/delivery tools to automate their testing and QA, all while deploying at an accelerated cadence.

However, no matter how hard we try to mitigate the risk of software delivery, almost all end-user software releases are strictly coupled with some form of code deployment. This means that companies must rely on testing and QA to identify all issues before a release hits production. It also means that companies primarily rely on version control systems or scraped together config files to control feature releases and mitigate risk.  For instance, many homegrown feature release systems rely on hard coded values read from config files.  These systems can work with a handful of configuration values, but accrue massive technical debt at scale and may require a full redeploy for any updates.

Once a release is in production, it is basically out in the wild.  Without proper controls, rolling back to previous versions becomes a code deployment exercise, requiring engineering expertise and increasing the potential for downtime. Continue reading “Powering Continuous Delivery With Feature Flags” »

30 May 2016

GlueCon 2016: Developer + Operations, Tools + Teams

GlueCon is a low-key, practitioner focused developer conference near Boulder, Colorado. Docker had been announced at a prior GlueCon, and so the conference has a well-deserved reputation as to where the next breakthrough will be announced. In a two-day conference with six tracks per day, you could “choose your own adventure” to learn about relevant tools for your domain of “Cloud, DevOps, Mobile, APIs, Big Data”, or get exposed to techniques outside your usual realm. I was amazed by how many people I thought I knew, then realized they looked familiar from their Twitter handle. However, if you’re looking for Dreamforce or the CES of Software, this is definitely NOT it – which I liked.

My favorite talk was Travis Vachon, Engineering Director of CircleCI on “DNA of Successful Development Teams”. The room was packed to the rafters, with people sitting on the side railings.  My main takeaway was

 Emmanuel Paraskakis, VP Product at Apiary.io talked about trying to get developers to write consistent APIs by providing them with a style guide full of “musts” and “should” for format. “No one read my 20-page style guide”. I think we’ve all been there. Continue reading “GlueCon 2016: Developer + Operations, Tools + Teams” »

13 May 2016

LaunchDarkly – Now Serving Billions of Feature Flags at Warp Speed

It’s been a busy spring for LaunchDarkly, the leading feature flag management company.  We’ve hit a huge milestone – we’re serving  Billions (with a B) feature flag events daily, for companies all over the world. 

LaunchDarkly helps companies with better DevOps by managing code deployment separately from business logic. We allow an entire product development organization to control who gets the right feature, when. Developer tool companies like CircleCI, Apiary and AppDirect use us to power their own DevOps.  But, LaunchDarkly isn’t just for developer tool companies to reduce risk and iterate quickly on feature release.  Across industries, companies like Ten-X, Behalf, Handshake, and TrustPilot integrated LaunchDarkly into their development and release cycles to move faster with higher quality. 

Why are companies moving so quickly to LaunchDarkly?  We offer true polyglot support for web and mobile applications. Have a Node.js web application with Go micro-services and a mobile app written in Swift?  We got you covered.  And, more importantly, we allow companies like CircleCI and Behalf to go beyond DevOps to transform the way they deliver software from idea to delivery by uniting product management, engineering, QA and operations. We offer end to end feature flag management to allow an entire product development organization to control who gets the right feature, when.

To help spread the word about the power of feature flagging, we also have awesome partners.  We were thrilled to participate in the Microsoft Build in March.  I was part of the Microsoft DevOps Keynote, showing our end to end integration with Visual Studio Team Services.  And we’re proud to power feature rollouts for effective DevOps at companies like: Continue reading “LaunchDarkly – Now Serving Billions of Feature Flags at Warp Speed” »