08 Jul

Feature Branching Using Feature Flags

Feature Branching Using Feature Flags Github LaunchDarkly

Feature branching allows developers to effectively collaborate around a central code base by keeping all changes to a specific feature in its own branch.  With the addition of feature flags, feature branching becomes even more powerful and faster by separating feature release management from code deployment.

The Pains of Long-Lived Branching

Traditionally, new development has been done via branching.  These branches typically live for months at a time, with silos of developers attempting to work in parallel.  Long-lived branches often become the “I owe you”  of Gitflow: “I promise to address these merge conflicts… later.”  The pain points that Git was supposed to address — merge conflicts, missing dependencies, and duplicate code — gradually emerge the longer a branch lasts.  We then concentrate all the pain of reworks and conflict remedies until the end, which arguably makes the pain much worse.  To avoid these long-lived branching issues, branching with feature flags enables devs to effectively manage short-lived feature branches and continuously deliver better software.

Feature Branching

To help coordinate development, engineering teams have adopted distributed version control systems (like GitHub and BitBucket).  These systems allow developers to collaborate within a central codebase and use branches to make updates or develop new features.  Feature branching, therefore, has become a staple of modern development cycles because they allow developers to function without encroaching on each other’s progress.

One of the shortcomings of feature branching is that feature release management is still tied to code deployments.  In isolation, feature branching forces engineers to manage software delivery within the confines of their version control system.  Non-technical users can’t (and shouldn’t) manage feature visibility and distribution for customers from GitHub.  It is currently not possible for developers to turn features on or off in real-time, as a way to manage risk or gradually introduce a new feature.

Feature Branching Without Feature Flags LaunchDarkly

This diagram illustrates how developers create feature branches to manage feature development, ensuring that the current build is always in a deployable state.  While every company’s branching structure will be different, the basic premise is that you branch from a repository to make a new feature.

Feature Branching Using Feature Flags

This is where the introduction of feature flags makes feature branching exceptionally powerful.  Feature flagging allows developers to take full control of their feature lifecycles independent of code deployments.  Application features can be enabled or disabled without requiring code to be reverted or redeployed.

This process is called feature flag driven development, where continuous delivery teams manage feature rollouts separate from code deployments.

Feature Branching With Feature Flags LaunchDarkly

Benefits

Feature flagging allows developers to take full control of their feature lifecycles without depending on code deployments.  When you merge a feature branch into master (production), it is already wrapped in a feature flag.  This allows you to deploy the feature “off” and then gradually roll it out to users.  It also allows you to quickly “kill” the feature if it is not working well, without having to redeploy.  Other benefits include:

  • Better team communication
      • Developers continuously collaborate on new releases, coordinate efforts, and ensure that everyone is moving forward in tandem.
  • Logical deployments to prevent blocking
      • Deploy code in logical chunks without resorting to long-lived branching.  Even if a feature is not fully ready, it can be flagged ‘off’, but still deployed.
  • Exposed dependencies
      • Short-lived branching allows you to manage dependencies by removing the ones that are unnecessary.
  • Faster development
      • Spend less time addressing merge conflicts and refactoring old code.  Spend more time delivering features that users want.
  • Risk mitigation
      • A feature can be flagged throughout the entire development process, from local, QA, staging, and production.  This means that it can be toggled on or off in isolation, without impacting other development environments.
  • Manageable code reviews
    • Because you are merging more often, code reviews become less tedious and merge conflicts less onerous.

Better, Together

Modern DVCS combines the benefits of short-lived branches and feature flags.  Tools like GitHub and Bitbucket enable continuous delivery and short-lived branching, while feature flags help make branching faster, less risky, and more manageable.
Therefore, feature flagging does not replace branching, it is complementary and makes it more powerful. Feature branching provides the flexibility to decide when and what to release, while feature flagging lets you take full control of the release itself.  Together, branching and flagging help you maintain and integrate short-lived branches faster and with less risk.

26 Jun

Staging Servers are Dead! Long Live a Staging Server

Earlier this year, I wrote about why staging servers should die – that they actually increase risk and time and decrease quality. I’ve been very pleased at the thoughtful comments and feedback I’ve gotten about why effective continuous delivery and DevOps means no staging server. The one that made me the happiest was “Dreams exist to become reality. Here is one I’d like to achieve at work.”

Here I’ll address the feedback I received, and what’s standing in the way of achieving this dream.

 

First, there was a large agreement that “waterfall deployment” with a staging server added time and risk to the launch process. Rapportive founder Sam Stokes agreed, saying “Staging environments are an evolutionary dead end. They get out of sync, slow you down, just get neglected.” https://twitter.com/samstokes/status/690689667306369024 Christan Deger, Autoscout Software Architect  said “staging environment, even in a completely automated cloud setup, requires constant effort and produces costs.” Dave Nolan gave a talk at Pipeline London https://vimeo.com/162633462 on #nostaging.

 

Here are the questions and concerns about the practicality and validity of the “#noStaging” Dream.

 

“Don’t we need to test major infrastructure changes in Staging?”

I’m not advocating pushing untested code willy nilly into production. I am advocating for through and complete automation, continuous integration and feature flagging, with the goal to get as quickly as possible to production. The reason to kill staging servers was said very well by Joseph Rustcio, Librato CTO & co-founder  “You think of everything in advance of how a user will use a feature, and you still miss half of them.” The purpose of DevOps is to move as fast as possible from code on a developers box to customer.

 

One great case study is Librato. Librato uses feature flags to wrap features, deploying their code and then ramping volume up and down, controlling risk. The allure of a staging server is that all systems can be thoroughly tested there, “guaranteeing” a painless final deployment. However, the moment where you push code to the real world is the scariest, because the real world NEVER matches staging. By using feature flagging, Librato could de-risk a new infrastructure project. They used “Branch-by-abstraction” to ramp on and off a new infrastructure. The old way would have been to push the entire new infrastructure at a slow time like 3 am, then when something went wrong, scramble to fix, precisely when memories and tempers are short due to sleep deprivation and key people might be asleep. With feature flags, Librato could ramp up volume in the real world, with real data. Ruscio says of their rollout with feature flags, “We never got paged at night for an issue with the new system, as we were only introducing it during the day when we were available. So all issues could be addressed while we were available. With feature flags, we could even dial back risk during lunch hour.”

 

“But we need a staging server for contractual reasons”

Having a staging server for contractual reasons is arguing for an artificial artifact. As Dave Farley, co-author of Continuous Delivery” says “it’s not done until it’s delivered to users”. Lamont Lucas, FastRobot co-founder, says “media and advertising companies want final approval before a feature goes live, generally from non-technical approvers. Flagging all users coming from a gateway lets you fulfill the contractual obligation (of showing a feature to them for approval) while preventing you from having an entire legacy/pointless staging setup.”

 

“What about performance testing?”

Sean Byrnes, CEO/Founder of Outlier and the Founder of Flurry, often had to “test that new features work in a production-like environment and don’t compromise the integrity of our systems”, as Flurry had millions of users worldwide.  However, Byrnes  continued, “You don’t have to load test EVERY feature to failure”.

The failure of most features it’s lack of load as there’s no customer interest. However for feature where you do really need to know the failure rate, an alternate server other than production can be spun up ephemerally for that reason. Just don’t call it “the staging server” – it’s a production-like load system spun up for a specific reason – to test load – and then shut down after it’s purpose is complete.

 

“Feature flags are only good for shallow UE changes, not for Microservices”.

Actually, micro services make using staging servers even more painful as many versions of different microservices might be running across staging and production. Deger says,  “ In a microservices architecture with lot of independent deployments going on, using a single staging environment would inadvertently test integrations with services in different versions, than what is currently in production. This gives a false sense of security. So we decided to not have a staging environment at all and find different ways to release features with confidence while only integrating in production.

We are constantly learning new techniques to deploy into production without a staging environment. There are many different ways of integrations that we need to care of, but overall it makes us faster and the testing and release experience is better.””

 

“Harbaugh is biased”

I advise that instead of a confusing, time consuming, expensive, redundant and ultimately unsuccessful staging server; development teams should use feature flags to move features to production and do their feature validation as quickly as possible on production. I’m not unbiased – I’m CEO of a feature flagging management platform, LaunchDarkly. However, I founded LaunchDarkly the same reason I publish articles, podcast , and give talks on DevOps: because I care deeply about the power of feature flagging to create better quality software and reduce risk. You don’t need LaunchDarkly to feature flag – it’s easy to get started with a simple config file or open source library. Deger, is the contributor to an open source framework for feature flagging and huge proponent of no staging servers.

 

“Let’s talk when you want to do serious software development”.

Just as waterfall development once seemed the only way to guarantee success, so did waterfall deployment until recently. The most innovative and fast moving companies like Netflix, Google, and Amazon have found that they can move faster if they’re more agile. And serious? I think of software development as fun, as exciting, as liberating, transforming – with software people are connected worldwide, interacting in incredible ways, and their lives are better. I hope software never becomes too serious for me.

08 Jun

How to Easily Sync Web & Mobile Experiences

Using feature flags to sync user experiences across multiple platforms

 

Becoming a VIP

Imagine you just signed up for Amazon’s VIP checkout experience.  In this hypothetical experience, you can click a VIP button that allows you to shop for brand new items not yet available to normal Amazon customers.  “This is awesome!” you say to yourself, as you view the VIP item list in your laptop’s browser.  

But now, you have to head to the bus and go to work, but you still want to browse for items using the Amazon app.  Because Amazon uses feature flags, your mobile experience is automatically personalized to include the VIP experience.  Cool!  Now, your web and mobile experiences are synced instantly.

Cross-Platform Personalization with Feature Flags

Feature flagging is a way to wrap features in conditionals (If/else statements) so that you can control the visibility of those features over time.  In other words, you can deploy a feature as ‘off’ and then turn it ‘on’ at a later time or you can gradually roll out a feature to select users.  

So, imagine you feature flag the VIP checkout.  You can create a rule that says: “Any user who opts into the VIP program will get TRUE for the feature and everyone else will get FALSE.”  Those users who are assigned the TRUE variation will see the VIP feature and those who are assigned the FALSE variation will not.

Mobile Web Feature Flags Feature Toggles LaunchDarkly

What is great about this method is that you can use this same feature flag to control the VIP checkout for both the web and mobile versions of an application.  

Some more benefits of cross-platform feature flagging include:

  • The ability to decide whether to release a cross-platform feature simultaneously or separately, with full control over who gets to see that feature. For example, web users might get access to a new search bar before mobile users do.
  • Real time personalization that allows users to opt-in to new features on one platform (like mobile) and have that personalization sync with another platform (like web)
  • Percentage rollouts that allow you to gradually release a feature to targeted users on different platforms, allowing you to assess user and performance feedback for each platform
  • A kill switch that lets you turn off poorly performing features for web and mobile, without having to redeploy
  • Subscription plan management using feature flags to bundle cross-platform features into different tiers (ex. to create Bronze, Gold, and VIP plans).

Summary

Cross-platform feature flagging is an easy way to deliver personalized and synchronized user experiences across different platforms.  Overall, it is paving a way for a new genre of user experience personalization, where companies can harness real time user feedback to customize features across different platforms.  This could usher in an era where users do not have to adapt their behavior to an app… the app will adapt to them.

19 May

Case study: Modernizing development & controlling rollout with LaunchDarkly

The challenge

The company, a leading online real estate marketplace, sought to bring its development up to date with modern processes. To accomplish this transformation, they hired management with experience at the most idolized mega tech companies.  The first thing on the manager’s agenda?  Move faster, with less risk by using a feature flagging system like the one they had built at their previous company.  

Just for some background – at this juncture, the team was doing three-week sprint cycles.  They had feature branches that they would develop every three weeks and then push out.  They would deploy five features at once and if anything went wrong, they’d have to roll it all back.  

Continue reading “Case study: Modernizing development & controlling rollout with LaunchDarkly” »

13 May

LaunchDarkly – Now Serving Billions of Feature Flags at Warp Speed

It’s been a busy spring for LaunchDarkly, the leading feature flag management company.  We’ve hit a huge milestone – we’re serving  Billions (with a B) feature flag events daily, for companies all over the world. 

LaunchDarkly helps companies with better DevOps by managing code deployment separately from business logic. We allow an entire product development organization to control who gets the right feature, when. Developer tool companies like CircleCI, Apiary and AppDirect use us to power their own DevOps.  But, LaunchDarkly isn’t just for developer tool companies to reduce risk and iterate quickly on feature release.  Across industries, companies like Ten-X, Behalf, Handshake, and TrustPilot integrated LaunchDarkly into their development and release cycles to move faster with higher quality. 

Why are companies moving so quickly to LaunchDarkly?  We offer true polyglot support for web and mobile applications. Have a Node.js web application with Go micro-services and a mobile app written in Swift?  We got you covered.  And, more importantly, we allow companies like CircleCI and Behalf to go beyond DevOps to transform the way they deliver software from idea to delivery by uniting product management, engineering, QA and operations. We offer end to end feature flag management to allow an entire product development organization to control who gets the right feature, when.

To help spread the word about the power of feature flagging, we also have awesome partners.  We were thrilled to participate in the Microsoft Build in March.  I was part of the Microsoft DevOps Keynote, showing our end to end integration with Visual Studio Team Services.  And we’re proud to power feature rollouts for effective DevOps at companies like: Continue reading “LaunchDarkly – Now Serving Billions of Feature Flags at Warp Speed” »

09 May

DevOps 2.0

DevOps2 LaunchDarkly

Decoupling feature rollout from code deployment and the rise of user-centered deployments

The Origin of DevOps

In 2008, Patrick Debois laid the foundations for DevOps at an Agile conference in Toronto.  He was trying to come up with a solution for the inherent conflicts between developers and system admins.   Both disciplines seemed to be at odds: developers wanted to release software more frequently, but system admins wanted to ensure stability, performance, and scalability.  While this conflict isn’t necessarily black and white, it highlighted the need for developers and system admins to no longer consider themselves as mutually exclusive roles, but rather as cross-functional partners.

A year later, Paul Hammond and John Allspaw gave a talk at the Velocity ‘09 conference that highlighted the necessity for cooperation between Dev and Ops.  This inspired Debois to coin the term “DevOps” (#DevOps), which quickly picked up momentum (and a fair share of controversy).

Now, with the proliferation of DevOpsDays meetups and DevOps communities around the world, the movement has accelerated.

Continue reading “DevOps 2.0” »