12 Apr 2017

How Spinnaker and Feature Flags Together Power DevOps

It’s very common for customers to be excited about both Spinnaker (continuous delivery platform) as well as feature flags. But wait? Aren’t they both continuous delivery platforms? Yes, they are both trying to solve the same pain points – the ability to quickly get code in a repeatable, non-breaking fashion, from the hands of the developers into the arms of hopefully excited end users, with a minimal amount of pain and heartache for everyone along the toolchain. But they solve different pain points:

  • Spinnaker helps you deploy functionality to clusters of machines.
  • Feature Flags help you connect those functionality to clusters of USERS.  

Spinnaker helps with “cluster management and deployment management”. With Spinnaker, it is possible to push out code changes rapidly, sometimes hundreds (if not thousands) of times a day. As Keanu Reeves would say “Whoa.” That’s great! All code is live in production! Spinnaker even has handy tools to run black/red deployments where traffic can be shunted from cluster to cluster based on benchmarks. Dude! For those who remember the “Release to Manufacturing” days where binaries had to be put on an FTP server (and hope that someone would install and download in the next quarter or so), code being live within a few minutes of being written is amazing. For those who remember “master disks” and packaged software, this is even more amazing.

Nevertheless, with dazzling speed comes another set of problems. All code can be pushed anytime. However, many times you do not want everyone to have access to the code – you want to run a canary release on actual users, not just machines. You might want QA to try your code in production, instead of a test server with partial data. If you’re a SaaS product, you might want your best customers to get access first to get their feedback. For call center software, you want to have an opportunity to test in a few call centers. You might want to have a marketing push in a certain country days (or weeks or months) after another country. You might want to fine-tune the feature with some power users, or see how new users react to a complicated use case. All of these scenarios can not be done at a server level. This is where feature flags come in. By feature flagging, you can gate off a code path, deploy using Spinnaker, and then use a feature flag to control actual access.

Together, Spinnaker and feature flagging make an amazing combination. You can quickly get code to “production”, and from there decide who gets it, when.

16 Feb 2017

Net Promoter Score + Feature Flags for canary releases

Canary releases are a DevOps best practice of pushing new features to a subset of your customers. By pushing to a subset, this group can provide early feedback, verify functionality, and act as “canaries” to your entire population. Originally, canary had a negative connotation – miners took canaries down into mines, as canaries were more sensitive to bad air. If the canary stopped singing, the miners knew it was time to skedaddle.

Surprisingly, being a canary can actually make your customers even bigger fans! I heard a great use case from a LaunchDarkly customer People.ai,  AI for managing sales teams, who would automatically push new features out to users who had an NPS of over a certain threshold. Net Promoter Score (NPS) rates how strongly would a customer recommend your solution, so initially, this seemed bizarre.  Why risk showing your best advocates functionality that might be unstable, early, or fragile? However, the people who are happiest users (likely to recommend) are actually very excited about being the first to see new functionality. They love feeling plugged-in to new development. And in addition, with LaunchDarkly feature flag management, if a feature isn’t doing well, the team can instantly turn it off, reverting to the old, stable experience.

I love hearing stories about how customers are using LaunchDarkly’s segmentation capabilities to release better software, quicker.


02 Dec 2015

In 2016, can DevOps keep pace with consumer expectations?

Consumer Expectations

Every morning, I roll out of bed at 8:01am and lazily reach for my phone to check my Apple News feed.   Sometimes, the app has the audacity to make me wait 2 full seconds before it refreshes my feed with 100 articles from dozens of sources around the web.

Unacceptable!  I want my news feed instantly.  I want it now.  I can feel myself get frustrated and I can feel my patience evaporate in a flash.  Then I take a step back: since when do I take personal offense to an application that doesn’t load instantly?  Why do I get flustered as if I was rear-ended by a car?

This is just a microcosm of current consumer expectations.   An iOS bug becomes a TechCrunch headline, a bad Facebook feature makes CNN’s front page, and an app that drains 0.1% more battery life becomes a human rights violation.

As a developer, imagine trying to adapt to these expectations.  Consumers want things to work instantly, smoothly, and intelligently.  They want everything to be perfect and work seamlessly or else they get frustrated, offended, and vocally upset.  “Ugh, why do I have to click TWO buttons to see this?”


Moving into 2016, what can developers do to meet or even exceed these expectations? Simply releasing software faster will not necessarily mitigate risk, nor will it necessarily lead to a better product.  The key question to address is “how can I adapt my development process to exceed consumer expectations?”

It doesn’t matter how wonderful your new feature is if it degrades your application’s performance.  Product-market fit no longer means that your product merely serves a market, but that it must meet that market’s expectations for performance.

How you release a feature becomes as important as the feature itself.   DevOps in 2016 should be the year of the incremental rollout, whereby assessing your application’s response to a new feature becomes a prerequisite for a launch.  I am not strictly referring to local or staged testing, but actual testing in production.

I recently published a piece on feature flag driven development and I feel that this practice of compartmentalized release is essential for managing risk and meeting consumer expectations.  By flagging a feature (i.e wrapping it in a condition), you can deploy it off and then incrementally turn it on for particular users, assessing performance feedback along the way.

LaunchDarkly Feature Flag Rollouts

Managing Scalability with Rollouts

Let’s look at an example.  You are a developer launching a new feature that will require you to process hundreds of additional requests per second.  A few hundred more?  That’s no problem – you’ve built the infrastructure to scale and handle that load.  But, what if all your users fall in love with the new feature?  Bombarding you with thousands of requests.  How do you manage this?

Imagine that you were able to roll out your feature live to 10% of your users and then 20%… 30%….  Each step becomes a testing benchmark where you assess performance feedback and can scale accordingly.

More importantly, you mitigate unanticipated performance degradation and meet the consumer expectation of seamless application performance.

implementing rollouts

Of course, feature flagged rollouts are not the saving grace for every feature launch or app update, but they’re fast becoming essential for DevOps as consumer expectations continue to raise the bar for performance.

Managing risk does not need to be a transformative operational process.  It can be easily achieved by flagging your features and gradually releasing them to your users.  After all, there’s no better way to get genuine feedback than testing in a real environment.

24 Aug 2015

Secret to Google’s Engineering Culture

Google is known for rock solid stability and iterating quickly on user feedback. One of Google’s secrets is canary releases to ensure stable, high quality products. A canary release is releasing features to some users (but not all). If the features aren’t successful, they are reverted to be fixed or removed. Google has an extremely sophisticated canary release process. Google has extensive unit tests that allow them to continuously integrate. After continuous integration, for major products like Google Mail, Google has a weekly release cycle.

Google’s first canary is google.com itself, as it has over 50,000 employees. Starting Monday and through the week, Google gradually rolls the new release internally to it’s own users, monitoring hundreds of different metrics on scalability and stability. Google’s employees act as canaries – squawking about any issues that come up. If core metrics are impacted significantly, the release is rolled back entirely. However, usually fixes are put in for critical issues and the release goes to the entire Google employee base.

On the next Monday, Google will take the release that had been rolled out to Google employees and canary release externally. Google will start showing more users the new release. Google continues to monitor for scalability and stability – using the external users as the true “staging” environment. Google’s products have a large enough base that they can even do sentiment analysis for keywords like “Gmail Sucks” on twitter. If there are spikes, Google will stop or roll back the new release. But usually by the end of the week, the entire release has been rolled out, and it’s time for the new new release.

The advantages of Google’s canary release approach is

  • stability – there’s always a release to roll back to
  • feedback – real world users give real world reactions

Google is large enough that they can never simulate or test all situations internally on an artificial staging server. By pushing releases to real users as canaries, Google ensures that they’re designing for the real world, not a laboratory. What enables Google to roll releases to real users is a deployment system that allows them to easily control who sees what features, and roll back easily. Want to be as smart and stable as Google? You could build your own canary deployment system – or you could use LaunchDarkly, which allows you to roll out features to your own users. LaunchDarkly even integrates with NewRelic so it’s easy to see metrics on your features.

15 Jul 2015

Feature flags, dark launches, and canary releases for all: LaunchDarkly first year in review

It’s been a year since we officially started full time on LaunchDarkly. Leading up to our official first day on July 14, 2014, John and I’d had been sharing ideas for years on how continuous delivery, agile and lean startups had changed the game for effective software development. Back when it was state of the art to release more than once a year, I remember having release parties. Now SaaS rules. Packaged, installed software is dying (or on it’s last gasps). The smart companies like Facebook, LinkedIn, Etsy and Netflix are releasing multiple times per day, and even hour, directly to their users. By iterating quicker and listening to their customers, these companies were delighting their users with more features. As well, developers were happier – it’s painful to build features for years on end, only to find out you’ve missed the mark.

Flickr first started talking about the feature flag/feature flipper pattern in 2009 as their key to engineering success. Facebook kicked off popularizing dark launches in 2009. Etsy, in 2011, noted how feature flags were helping them scale. After Instagram was acquired by Facebook, they adopted Facebook’s practice of canary releases.

Large companies like Facebook and DropBox could afford to build and maintain a dedicated framework for their feature flags, dark launch software and canary releases. Facebook calls their feature flag and experimentation framework: Gatekeeper; DropBox: Gandalf (“none shall pass”). But everyone else who wanted the powerful ability to deploy multiple times per day, control who saw what features, and move fast and ship things had three choices – to build their own expensive infrastructure in house, to ship and hope for the best (the ostrich deploy) or to sit out the continuous delivery movement. John and I saw an opportunity to be “Gandalf for everyone”- dark launch software as a service. LaunchDarkly would let everyone feature flag, dark launch, canary release, and use the continuous deployment tools of a big player, at a fraction of the cost.

So on July 8th, 2014, John checked in his first code on what would become LaunchDarkly. Our official first day of work was July 15th, when we went to work together for the first time. The year has gone by so quickly – we have our first customers, we’ve been joined by our engineers, Alexis Georges and Patrick Kaeding, and we even had our first Dark Launch meetup. What’s next? Continuing to iterate on our features, listening to our customers – continuing to Launch Dark!


09 Jul 2015

Canary release is the new beta

Are canary releases the new beta? What does beta even mean? Sean Murphy recently tweeted me:

Wow. Was Sean right? When I was an Engineering Manager at Vignette, I’d run beta programs for our new releases. The beta programs had a dual purpose. First, we wanted to get feedback on the stability and validity of our features. But the beta also fed marketing with happy reference customers for our launch announcement. Customers liked being part of a beta because it gave them early access to features they had been waiting for, as well as an opportunity to influence product direction.

What had changed? The word beta has been overloaded to mean “we’re not entirely ready for prime time, so please be patient”. Gmail was in beta for five years! At TripIt, we had a beta tag for multiple years.

Canary release – exposing features to some subset of users (whether it be opt-in, random rollout, or specific segments) is now used to describe what was once a beta.

  • Microsoft: In development of Windows 10, Microsoft used “canary” releases to test with internal users within Microsoft. Gabe Aul, who leads the Data & Fundamentals Team in the Operating Systems Group (OSG), said “our Canary ring probably sees 2X-3X as many builds as OSG because we catch problems in Canary and don’t push to OSG.”
  • Instagram: “Using ‘canary’ releases, updates go out to a subset of users at first, limiting the ability of buggy software to do damage.” Mike Krieger, Instagram co-founder and CTO, said he uses canary releases because “If stuff blows up it affects a very small percentage of people”.
  • Google: For Chrome, Google offers Chrome Canary, which it labels with “Get on the bleeding edge of the web, Google ChromeCanary has the newest of the new Chrome features. Be forewarned: it’s designed for developers and early adopters, and can sometimes break down completely.”

So yes, canary is the new beta.