23 May

Risk Elimination and The LaunchDarkly Value-Add

My first week at LaunchDarkly brought me out of the shadows in a hurry.  It began at an offsite strategy session at DFJ, our lead investor, where I learned valuable details about Waterfall vs. Agile software development methodologies. I also gained important insights into a key industry trend affecting the development community: the transition from Waterfall to Continuous Integration/Continuous Delivery (CI/CD).

What I’ve learned as a marketer has deepened my appreciation for what makes the LaunchDarkly solution so unique.

For starters, there are three main categories of customers who will benefit from partnering with LaunchDarkly:

  1. Companies interested in switching from Waterfall to CI/CD
  2. Companies currently switching/recently switched from Waterfall to CI/CD but not yet feature flagging
  3. Companies that are currently engaged in CI/CD, and using a homegrown feature management system

What’s clear is that all three of these customer segments experience different challenges. But all fall into to what our VP of Product and Platform, Adam Zimman, calls “The Risk Gap.”

What is The Risk Gap?

In software development, there is inherent risk in launching new releases. Risk in this case can be broken down into two categories:

  • Risk of losing product value
  • Risk of losing time

The longer it takes an engineering team to launch a new software release, the greater the risk of feature obsolescence. Another risk factor is competitor time to market; those companies that don’t enjoy “first mover advantage” can suffer from demoralized developers who lose interest because they can’t ship quickly enough.

The Risk Gap also means that there is greater operational risk associated with feature releases that carry greater value. The more value associated with a feature update, the greater the risk to your Ops team, because of changes made to your code base.

The Risk Gap is closely linked to the Iron Triangle concept that suggests the following:  while teams should strive to release high value features at a quick pace, the reality is that they’re often forced to pick one or the other (speed vs. quality).

The Iron Triangle mantra is “Fast, good, or cheap. Pick two.”

Let’s see how this affects the three customer categories who will benefit from using LaunchDarkly by examining the Risk Gap/Iron Triangle framework.

CategoryPainDoes Have Does Not Have
Companies interested in switching from Waterfall to CI/CDTakes Dev team a long time to launch releases.-High Quality
-Low Cost - traditional Waterfall methodology
Fast Delivery
Companies switching/recently switched from Waterfall to CI/CD but not yet feature flaggingQuality of releases is at risk.-Fast Delivery via CI/CD
-Lower Costs - not using a feature management platform
High Quality
Companies doing CI/CD + using a homegrown feature management systemA homegrown feature management system is costly to develop and maintain.-Fast Delivery - quick release cycle
-High Quality - continuous feedback loop
Lowest Cost

Each customer category is missing one of the three components of the Iron Triangle: either quality, speed, or lowest cost.

LaunchDarkly’s value-add

LaunchDarkly exists to close the Risk Gap – enabling the largest software engineering teams in the world to responsibly employ the CI/CD methodology, accelerate development cycles, eliminate the risk associated with large releases, and cut costs of developing/maintaining homegrown feature management systems.

For the first time, you don’t have to make tradeoffs with LaunchDarkly.

When you combine the great team here, a revolutionary product, and the opportunity to learn from brilliant minds every day, I am very much so looking forward to bringing our product to market.

26 Jun

Staging Servers are Dead! Long Live a Staging Server

Earlier this year, I wrote about why staging servers should die – that they actually increase risk and time and decrease quality. I’ve been very pleased at the thoughtful comments and feedback I’ve gotten about why effective continuous delivery and DevOps means no staging server. The one that made me the happiest was “Dreams exist to become reality. Here is one I’d like to achieve at work.”

Here I’ll address the feedback I received, and what’s standing in the way of achieving this dream.

 

First, there was a large agreement that “waterfall deployment” with a staging server added time and risk to the launch process. Rapportive founder Sam Stokes agreed, saying “Staging environments are an evolutionary dead end. They get out of sync, slow you down, just get neglected.” https://twitter.com/samstokes/status/690689667306369024 Christan Deger, Autoscout Software Architect  said “staging environment, even in a completely automated cloud setup, requires constant effort and produces costs.” Dave Nolan gave a talk at Pipeline London https://vimeo.com/162633462 on #nostaging.

 

Here are the questions and concerns about the practicality and validity of the “#noStaging” Dream.

 

“Don’t we need to test major infrastructure changes in Staging?”

I’m not advocating pushing untested code willy nilly into production. I am advocating for through and complete automation, continuous integration and feature flagging, with the goal to get as quickly as possible to production. The reason to kill staging servers was said very well by Joseph Rustcio, Librato CTO & co-founder  “You think of everything in advance of how a user will use a feature, and you still miss half of them.” The purpose of DevOps is to move as fast as possible from code on a developers box to customer.

 

One great case study is Librato. Librato uses feature flags to wrap features, deploying their code and then ramping volume up and down, controlling risk. The allure of a staging server is that all systems can be thoroughly tested there, “guaranteeing” a painless final deployment. However, the moment where you push code to the real world is the scariest, because the real world NEVER matches staging. By using feature flagging, Librato could de-risk a new infrastructure project. They used “Branch-by-abstraction” to ramp on and off a new infrastructure. The old way would have been to push the entire new infrastructure at a slow time like 3 am, then when something went wrong, scramble to fix, precisely when memories and tempers are short due to sleep deprivation and key people might be asleep. With feature flags, Librato could ramp up volume in the real world, with real data. Ruscio says of their rollout with feature flags, “We never got paged at night for an issue with the new system, as we were only introducing it during the day when we were available. So all issues could be addressed while we were available. With feature flags, we could even dial back risk during lunch hour.”

 

“But we need a staging server for contractual reasons”

Having a staging server for contractual reasons is arguing for an artificial artifact. As Dave Farley, co-author of Continuous Delivery” says “it’s not done until it’s delivered to users”. Lamont Lucas, FastRobot co-founder, says “media and advertising companies want final approval before a feature goes live, generally from non-technical approvers. Flagging all users coming from a gateway lets you fulfill the contractual obligation (of showing a feature to them for approval) while preventing you from having an entire legacy/pointless staging setup.”

 

“What about performance testing?”

Sean Byrnes, CEO/Founder of Outlier and the Founder of Flurry, often had to “test that new features work in a production-like environment and don’t compromise the integrity of our systems”, as Flurry had millions of users worldwide.  However, Byrnes  continued, “You don’t have to load test EVERY feature to failure”.

The failure of most features it’s lack of load as there’s no customer interest. However for feature where you do really need to know the failure rate, an alternate server other than production can be spun up ephemerally for that reason. Just don’t call it “the staging server” – it’s a production-like load system spun up for a specific reason – to test load – and then shut down after it’s purpose is complete.

 

“Feature flags are only good for shallow UE changes, not for Microservices”.

Actually, micro services make using staging servers even more painful as many versions of different microservices might be running across staging and production. Deger says,  “ In a microservices architecture with lot of independent deployments going on, using a single staging environment would inadvertently test integrations with services in different versions, than what is currently in production. This gives a false sense of security. So we decided to not have a staging environment at all and find different ways to release features with confidence while only integrating in production.

We are constantly learning new techniques to deploy into production without a staging environment. There are many different ways of integrations that we need to care of, but overall it makes us faster and the testing and release experience is better.””

 

“Harbaugh is biased”

I advise that instead of a confusing, time consuming, expensive, redundant and ultimately unsuccessful staging server; development teams should use feature flags to move features to production and do their feature validation as quickly as possible on production. I’m not unbiased – I’m CEO of a feature flagging management platform, LaunchDarkly. However, I founded LaunchDarkly the same reason I publish articles, podcast , and give talks on DevOps: because I care deeply about the power of feature flagging to create better quality software and reduce risk. You don’t need LaunchDarkly to feature flag – it’s easy to get started with a simple config file or open source library. Deger, is the contributor to an open source framework for feature flagging and huge proponent of no staging servers.

 

“Let’s talk when you want to do serious software development”.

Just as waterfall development once seemed the only way to guarantee success, so did waterfall deployment until recently. The most innovative and fast moving companies like Netflix, Google, and Amazon have found that they can move faster if they’re more agile. And serious? I think of software development as fun, as exciting, as liberating, transforming – with software people are connected worldwide, interacting in incredible ways, and their lives are better. I hope software never becomes too serious for me.

15 Jul

Feature flags, dark launches, and canary releases for all: LaunchDarkly first year in review

It’s been a year since we officially started full time on LaunchDarkly. Leading up to our official first day on July 14, 2014, John and I’d had been sharing ideas for years on how continuous delivery, agile and lean startups had changed the game for effective software development. Back when it was state of the art to release more than once a year, I remember having release parties. Now SaaS rules. Packaged, installed software is dying (or on it’s last gasps). The smart companies like Facebook, LinkedIn, Etsy and Netflix are releasing multiple times per day, and even hour, directly to their users. By iterating quicker and listening to their customers, these companies were delighting their users with more features. As well, developers were happier – it’s painful to build features for years on end, only to find out you’ve missed the mark.

Flickr first started talking about the feature flag/feature flipper pattern in 2009 as their key to engineering success. Facebook kicked off popularizing dark launches in 2009. Etsy, in 2011, noted how feature flags were helping them scale. After Instagram was acquired by Facebook, they adopted Facebook’s practice of canary releases.

Large companies like Facebook and DropBox could afford to build and maintain a dedicated framework for their feature flags, dark launch software and canary releases. Facebook calls their feature flag and experimentation framework: Gatekeeper; DropBox: Gandalf (“none shall pass”). But everyone else who wanted the powerful ability to deploy multiple times per day, control who saw what features, and move fast and ship things had three choices – to build their own expensive infrastructure in house, to ship and hope for the best (the ostrich deploy) or to sit out the continuous delivery movement. John and I saw an opportunity to be “Gandalf for everyone”- dark launch software as a service. LaunchDarkly would let everyone feature flag, dark launch, canary release, and use the continuous deployment tools of a big player, at a fraction of the cost.

So on July 8th, 2014, John checked in his first code on what would become LaunchDarkly. Our official first day of work was July 15th, when we went to work together for the first time. The year has gone by so quickly – we have our first customers, we’ve been joined by our engineers, Alexis Georges and Patrick Kaeding, and we even had our first Dark Launch meetup. What’s next? Continuing to iterate on our features, listening to our customers – continuing to Launch Dark!

 


LAUNCHDARKLY HELPS YOU BUILD BETTER SOFTWARE FASTER WITH FEATURE FLAGS AS A SERVICE. START YOUR FREE TRIAL NOW.