Velocity is a great conference for ideas about where software and companies are going in the future. And I got a chance to talk with Mike Hendrickson about LaunchDarkly and feature management in that context.
We think that feature management is a best practice for anyone who is trying to do continuous integration and deployment. Feature management gives you a way to test features in production quietly before making them visible to everyone. At the end of this interview we talked about what I think will happen in the future—I’m excited about the idea of giving users the exact page or experience they want and need.
If you’re interested in learning a bit more about feature management or LaunchDarkly, take a few minutes to watch this video.
We recently looked at how some well-known companies beta test. Specifically we looked at groups that test in production, and do it well. As you know, testing in production is one of the best ways to find bugs and get solid feedback from your users. While some may shy away from this because of the risks involved, there are ways to mitigate risk and do it right. So this time we want to share how we beta at LaunchDarkly.
It’s no surprise that we dogfood at LaunchDarkly. Using feature flags within our development cycle is a straightforward process. We often push features directly into our production environment and safely test prior to allowing user access. When it’s time to beta test with users we can update the setting on the appropriate flag and get user feedback quickly. And of course, if we ever need to we can instantly turn features off.
Deciding Which Things to Build
When we’re thinking about new features to implement, we have our own ideas of which direction our product should go, but we also consider inbound requests. This can be from support tickets, questions from potential customers, or conversations with existing customers. Bottom line is we want to build a product that serves our customers, and so we do our best to listen to what they want.
Once we identify a feature we’d like to build—whether it was our own idea or a customer request—we’ll share it out to see if other customers are also interested. This is an important part of our beta testing process, because once the feature is in production and we’re ready to test it, these are the people we want to circle back with for beta testing.
Testing in Production
When it’s time to test, we test with actual end users in production. Our feature management platform allows us to turn features on for specific users. We can specify individual users, or we can expose users by attribute, like region (everyone in Denver)—and we can instantly turn them off at any time.
Because we’re testing in production, we don’t have to have an isolated environment or separate account. For those customers who showed interest, and agreed to participate in beta testing, we turn the features on in their production accounts.
Typically we beta for two weeks, sometimes as long as a month. As mentioned before, since we know which customers are interested in the feature, we can go back to them and have them test it. These are the users who already know they want this functionality, so we want to be sure it fits (or exceeds!) their expectations. And of course we want to make the most of this time, so it’s important we actually get feedback. We find that those who have asked for the feature are eager to let us know how things are working. We make a point of also following up with those who don’t proactively offer feedback—we want to hear from everyone!
While we’re testing and getting feedback, we’re taking all this information in and improving the feature before rolling it out to everyone else. When we feel confident we have something that’s ready to be shared, we’ll begin a percentage rollout to the rest of our users.
Using feature flags around features within our development cycles allows us to mitigate risk by pushing out small, incremental changes at a time. As you can see, this also enables us to beta test quickly and safely. If there are major bugs, we’re more likely to identify them early on before affecting our all of our customers.
“Embrace failure. Chaos and failure are your friends. The issue is not if you will fail, it is when you will fail, and whether you will notice.” -Charity Majors
Right now we’re currently in beta for scoped access tokens and a new faster .net SDK. Let us know if you’d like to take a look at it early, we’d love to hear what you think.
We all know beta testing is important—not just for understanding your customers’ needs, but also for stability and security. Every time you do a launch you are essentially asking: “Are there bugs? Is there feedback?” Both with the goal of making your product better.
Testing in production will give you the most information about the success of your new functionality. And because feature flags help separate deployment from release, they make such testing safe and easy. When it comes to beta testing, a lot of the top companies tend to adhere to a similar paradigm—test early, test often, and do it in your production environment.
So how do companies have smooth and simple transitions from alpha to beta testing, and then to full rollout? Read on to learn how top companies are approaching their beta testing using deployment tools with feature flags providing links out to more in-depth descriptions.
But before we get started, here’s a quick terminology review. Pete Hodgson refers to this use of feature flags for betas as “permissioning toggles.” Also known as a “canary launch,” this is often random like a percentage rollout. A set group, or “champagne brunch,” releases to internal users or another section or group.
6 Approaches to Product Launching
#1 Facebook is the prime example of dark launching. Their release management has to be impeccable to operate at such massive scale. Their betas are often up to million users or more.
“Although we push to production only once a week, it’s still important to test the code early in real-world settings so that engineers can get quick feedback. We make mobile release candidates available every day for canary users, including 1 million or so Android beta testers.”
#2 Hootsuite gives a typical rollout pattern for its features—starting internally and then slowly exposing to a larger audience.
Push new code then:
– Dark launch to yourself or your team to test
– Launch to the whole Hootsuite organization
– 10% of all users
– Watch graphs
– Simple means of rollback if necessary
#3 Etsy calls feature flags “config flags,” and gives a lot of credit for their process to Flickr.
“Key system-level and business level metrics (like checkout/listing/registration/sign-in rates) are projected on screens in the office and we have a number of internal dashboards that the team uses (we mainly use Ganglia and Graphite). We also have lots of switches and knobs to help us roll features out to percentages of users and ramp them up slowly, or quickly. Features are used and tested by us here at Etsy for some period of time before they are rolled out publicly.”
#4 Beta can also apply to back-end rollouts. Instagram does canary deployments to a subset of servers, using feature flags as a continuous delivery tool. It’s important for continuous delivery to perform these tests, which are key in helping them avoid failed deployments.
#5 Niantic’s Pokemon Go betas are well known and rabidly tracked by its fans. They famously roll out by region—a field test in Japan here, a limited beta in Australia, and then something in New Zealand. Sometimes these betas for features are invite-only. Here’s a write up of how they approached the rollout of the game Ingress.
#6 GoPro released their GoPro Plus product early using feature flags. By breaking the larger release into smaller features with their own testing timelines, they were able to iterate and improve continuously. The video below walks through the technology they used and the timeline from dogfood to a “big bang” marketing announcement.
“At GoPro you can kind of tell we don’t things lightly. We want to do big announcements and we want to come out with great products…we actually had smaller features that would go out, and then go for alpha testing and beta testing along the way. Shortly after March, we actually had most of the applications done from a core feature standpoint, but we kept iterating and improving those core features that we knew we were going to launch with.”
Controlling Your Rollout Like a Boss
Did you notice some trends there? These larger companies are using beta testing to do one of the following:
Testing in production with feature flags
Ability to release early and test small functionalities before a broader release
Internal tests that easily become external canaries
As more companies start to use feature management, these incremental rollouts are not the headaches they once were. Companies can be safer and smarter with how and when they expose features to their end users.
If you want to get started with feature flagging, check out featureflags.io a resource we made for the community to learn best practices.
You are a new Director of Engineering at an enterprise company. The company has just moved your product to the cloud and is hosting in Microsoft Azure. You come from a tech startup where you ran all of your software in the cloud and focused on building product. So naturally you have just implemented instrumented monitoring with HoneyComb, Kubernetes to manage your containers, and you’re thinking about leveraging serverless architecture.
You moved to this established enterprise company because they are providing technology to the healthcare industry and you are passionate about this space. They want you because you are good.
Now you face a challenge—how can I bring the good things from my past role and pair them with the security standards and compliance in my new role? Well, let’s first think about what the good things from your past role entails. Your previous team:
Deployed 5 times a day
Used feature branching
Feature flag first development
Implemented continuous integration
You realize you used 3rd party software for most of these good things, and had homegrown for others. You decide your first project is to research what software the new team is using, has built, and what software you will need to build and buy.
Security and the cloud.
Now, before we dive in, let’s consider the security standards and compliance you need to think about in your new role:
Where is my data stored?
What data is being sent over to the third party—is it PII?
Where connections are made to third parties?
How can I set different access control?
How it will work when connection fails-redundancy?
Is it HIPAA compliant?
How is all this audited?
Like your new company, many companies are just starting to move key operations to the cloud. This is SCARY. Moving into Azure will allow your company to free up floor space, add more server redundancy, easily scale up and down, only pay for what you use, and focus resources on revenue generating activities— security is still your responsibility. And though these are all great things, you are not a security company.
Now you are tasked with bringing in these good processes, and they include 3rd party software providers. Products in this space are inherent in your development lifecycle and very close to your core application, but you need to ensure they are secure.
As we all know, most software providers in this space are cool tech companies in The Valley, far and isolated from the realities of your secure enterprise world. You look at on-premise options. And you remember that you’re driving change in the Healthcare space, not looking to manage infrastructure. The cloud options require a significant audit that is time consuming. A SaaS provider carries an ongoing risk that requires you store some data on 3rd party servers, which is a non-starter in Healthcare.
Is there a middle ground?
There is one more option many software providers offer: Private SaaS, also known as a managed instance, Dedicated VPC, or private instance. You surmise if they do not have on-prem, private SaaS, or a really really good security team, then it’s not likely their team is accustomed to working on enterprise challenges.
What exactly is a private SaaS offering? This refers to a dedicated single tenant, cloud software where the vendor manages the infrastructure.
Why choose a private instance?
Single Tenant—You will have a dedicated set of infrastructure contained within a VPC. This eliminates the risk of noisy neighbors.
Data Storage—Data can be stored in your AWS account or in an isolated section of the vendor’s AWS account. This allows flexibility, if you are in the EU, vendor could spin up the instance in AWS in the EU.
Residency—If you have a preference on where the instance is located or need to ensure proximity.
Compliance—If you have additional security or compliance regulations, a private instance can be customized to fit your needs.
Change Cadence—If you need to know when the software will be updated, you will have better insight and greater flexibility with a private instance.
Integrations—If you have custom tools, integrations can be built.
You have narrowed down a few software providers for each job you are trying to solve (continuous integration, branching, feature flagging). You understand the value, the costs, and the security implications. You have chosen to stay in the cloud, which makes your infrastructure team happy. You have chosen to use private instances and SaaS where it makes sense, which makes your security team happy. And you have the tools you need to bring the good things into your new role, so you are happy.
Now you can focus on helping your company deliver product faster, eliminate risk in your release process, speed up your product feedback loop, and do it all securely.
Start? This is just the next step in the journey (image credit: Andrew Lipson)
I recently wrapped up my first official full week at LaunchDarkly. Although, I’ve been working with the team as an advisor/consultant for a number of months. Over the past year I have been advising and consulting with various start-ups looking for a good fit. I would often remark to folks that I was “company dating.”
More like introductions from friends than tinder. (Photo credit: Reddit post)
Honestly, I have very little dating experience. My wife and I met my 2nd (her 1st) year in college and haven’t looked back. Similar on the job side, I started at EMC a year after graduating and then was the first internal transfer to VMware. For almost 15 years I enjoyed the stability and resources of a large company. But, then I started to feel the need to grow and participate in the changing landscape I saw in software development trends.
Since leaving VMware I have spent a lot of time thinking about the gap between the old school development frameworks (e.g. waterfall) and newer practices (e.g. agile, scrum, continuous deployment). Tools like git, continuous integration, and automation have radically changed how we release. At EMC and VMware we measured our releases in years, or sometimes months (the same way a parent refers to their 22 month-old toddler). Compared to GitHub where we released multiple times a day.
This whole cloud thing is likely just a fad. (Photo credit: Twitter)
Recently, I’ve started to bucket these tools into three phases of for software development: Concept, Launch, and Control. I’m working on a blog series to discuss each of these in-depth, but this framework is what got me excited about LaunchDarkly. Feature management, while not the shiniest tool, provides the foundation for eliminating risk and delivering value as teams push to move faster and to be more reliable.
In addition to my passion for our product, this intelligent team, my carless commute, I did have one additional objective: to be a part of a diverse and equitable company. Not simply an organization that accepts diversity, but one that actively pursues a more diverse and inclusive team as a imperative for building better products and services. So far a great start to my next long-term relationship.
My first week at LaunchDarkly brought me out of the shadows in a hurry. It began at an offsite strategy session at DFJ, our lead investor, where I learned valuable details about Waterfall vs. Agile software development methodologies. I also gained important insights into a key industry trend affecting the development community: the transition from Waterfall to Continuous Integration/Continuous Delivery (CI/CD).
What I’ve learned as a marketer has deepened my appreciation for what makes the LaunchDarkly solution so unique.
For starters, there are three main categories of customers who will benefit from partnering with LaunchDarkly:
Companies interested in switching from Waterfall to CI/CD
Companies currently switching/recently switched from Waterfall to CI/CD but not yet feature flagging
Companies that are currently engaged in CI/CD, and using a homegrown feature management system
What’s clear is that all three of these customer segments experience different challenges. But all fall into to what our VP of Product and Platform, Adam Zimman, calls “The Risk Gap.”
What is The Risk Gap?
In software development, there is inherent risk in launching new releases. Risk in this case can be broken down into two categories:
Risk of losing product value
Risk of losing time
The longer it takes an engineering team to launch a new software release, the greater the risk of feature obsolescence. Another risk factor is competitor time to market; those companies that don’t enjoy “first mover advantage” can suffer from demoralized developers who lose interest because they can’t ship quickly enough.
The Risk Gap also means that there is greater operational risk associated with feature releases that carry greater value. The more value associated with a feature update, the greater the risk to your Ops team, because of changes made to your code base.
The Risk Gap is closely linked to the Iron Triangle concept that suggests the following: while teams should strive to release high value features at a quick pace, the reality is that they’re often forced to pick one or the other (speed vs. quality).
The Iron Triangle mantra is “Fast, good, or cheap. Pick two.”
Let’s see how this affects the three customer categories who will benefit from using LaunchDarkly by examining the Risk Gap/Iron Triangle framework.
Does Not Have
Companies interested in switching from Waterfall to CI/CD
Takes Dev team a long time to launch releases.
-Low Cost - traditional Waterfall methodology
Companies switching/recently switched from Waterfall to CI/CD but not yet feature flagging
Quality of releases is at risk.
-Fast Delivery via CI/CD
-Lower Costs - not using a feature management platform
Companies doing CI/CD + using a homegrown feature management system
A homegrown feature management system is costly to develop and maintain.
Each customer category is missing one of the three components of the Iron Triangle: either quality, speed, or lowest cost.
LaunchDarkly exists to close the Risk Gap – enabling the largest software engineering teams in the world to responsibly employ the CI/CD methodology, accelerate development cycles, eliminate the risk associated with large releases, and cut costs of developing/maintaining homegrown feature management systems.
For the first time, you don’t have to make tradeoffs with LaunchDarkly.
When you combine the great team here, a revolutionary product, and the opportunity to learn from brilliant minds every day, I am very much so looking forward to bringing our product to market.