18 Oct 2017

How We Beta Test at LaunchDarkly

Photo by Alex Holyoake on Unsplash

We recently looked at how some well-known companies beta test. Specifically we looked at groups that test in production, and do it well. As you know, testing in production is one of the best ways to find bugs and get solid feedback from your users. While some may shy away from this because of the risks involved, there are ways to mitigate risk and do it right. So this time we want to share how we beta at LaunchDarkly.

It’s no surprise that we dogfood at LaunchDarkly. Using feature flags within our development cycle is a straightforward process. We often push features directly into our production environment and safely test prior to allowing user access. When it’s time to beta test with users we can update the setting on the appropriate flag and get user feedback quickly. And of course, if we ever need to we can instantly turn features off.

Deciding Which Things to Build

When we’re thinking about new features to implement, we have our own ideas of which direction our product should go, but we also consider inbound requests. This can be from support tickets, questions from potential customers, or conversations with existing customers. Bottom line is we want to build a product that serves our customers, and so we do our best to listen to what they want.

Once we identify a feature we’d like to build—whether it was our own idea or a customer request—we’ll share it out to see if other customers are also interested. This is an important part of our beta testing process, because once the feature is in production and we’re ready to test it, these are the people we want to circle back with for beta testing.

Testing in Production

When it’s time to test, we test with actual end users in production. Our feature management platform allows us to turn features on for specific users. We can specify individual users, or we can expose users by attribute, like region (everyone in Denver)—and we can instantly turn them off at any time.

Because we’re testing in production, we don’t have to have an isolated environment or separate account. For those customers who showed interest, and agreed to participate in beta testing, we turn the features on in their production accounts.

Typically we beta for two weeks, sometimes as long as a month. As mentioned before, since we know which customers are interested in the feature, we can go back to them and have them test it. These are the users who already know they want this functionality, so we want to be sure it fits (or exceeds!) their expectations. And of course we want to make the most of this time, so it’s important we actually get feedback. We find that those who have asked for the feature are eager to let us know how things are working. We make a point of also following up with those who don’t proactively offer feedback—we want to hear from everyone!

While we’re testing and getting feedback, we’re taking all this information in and improving the feature before rolling it out to everyone else. When we feel confident we have something that’s ready to be shared, we’ll begin a percentage rollout to the rest of our users.

Embracing Failure

Using feature flags around features within our development cycles allows us to mitigate risk by pushing out small, incremental changes at a time. As you can see, this also enables us to beta test quickly and safely. If there are major bugs, we’re more likely to identify them early on before affecting our all of our customers.

“Embrace failure. Chaos and failure are your friends. The issue is not if you will fail, it is when you will fail, and whether you will notice.” -Charity Majors

Right now we’re currently in beta for scoped access tokens and a new faster .net SDK. Let us know if you’d like to take a look at it early, we’d love to hear what you think.

14 Sep 2017

Beta Testing with Feature Toggles: Testing in Production Like a Pro

We all know beta testing is important—not just for understanding your customers’ needs, but also for stability and security. Every time you do a launch you are essentially asking: “Are there bugs? Is there feedback?” Both with the goal of making your product better.

Testing in production will give you the most information about the success of your new functionality. And because feature flags help separate deployment from release, they make such testing safe and easy. When it comes to beta testing, a lot of the top companies tend to adhere to a similar paradigm—test early, test often, and do it in your production environment.

So how do companies have smooth and simple transitions from alpha to beta testing, and then to full rollout? Read on to learn how top companies are approaching their beta testing using deployment tools with feature flags providing links out to more in-depth descriptions.

But before we get started, here’s a quick terminology review. Pete Hodgson refers to this use of feature flags for betas as “permissioning toggles.” Also known as a “canary launch,” this is often random like a percentage rollout. A set group, or “champagne brunch,” releases to internal users or another section or group.


6 Approaches to Product Launching

#1 Facebook is the prime example of dark launching. Their release management has to be impeccable to operate at such massive scale. Their betas are often up to  million users or more.

“Although we push to production only once a week, it’s still important to test the code early in real-world settings so that engineers can get quick feedback. We make mobile release candidates available every day for canary users, including 1 million or so Android beta testers.”

Read their article on Rapid Release at Massive Scale to learn more about how they do continuous delivery at scale.

#2 Hootsuite gives a typical rollout pattern for its features—starting internally and then slowly exposing to a larger audience.

Typical
Push new code then:
– Dark launch to yourself or your team to test
– Launch to the whole Hootsuite organization
– 10% of all users
– Watch graphs
– 50%
– 100%
– Simple means of rollback if necessary

Check out Bill Monkman’s full deck on dark launching here.

#3 Etsy calls feature flags “config flags,” and gives a lot of credit for their process to Flickr.

“Key system-level and business level metrics (like checkout/listing/registration/sign-in rates) are projected on screens in the office and we have a number of internal dashboards that the team uses (we mainly use Ganglia and Graphite). We also have lots of switches and knobs to help us roll features out to percentages of users and ramp them up slowly, or quickly. Features are used and tested by us here at Etsy for some period of time before they are rolled out publicly.”

They have custom built a feature flagging API, “Feature API” to enable this. Some of the bucketing they use include: admin, internal, users, groups.Read more about Etsy’s deployment practices and check out their Feature API on GitHub.

#4 Beta can also apply to back-end rollouts. Instagram does canary deployments to a subset of servers, using feature flags as a continuous delivery tool. It’s important for continuous delivery to perform these tests, which are key in helping them avoid failed deployments.

But Instagram hasn’t always had this system. Read here to learn how they evolved from a “mish-mash of manual steps and scripts” to a system they could depend on. And check this out if you want more recipes for database migration with feature flags.

#5 Niantic’s Pokemon Go betas are well known and rabidly tracked by its fans. They famously roll out by region—a field test in Japan here, a limited beta in Australia, and then something in New Zealand. Sometimes these betas for features are invite-only. Here’s a write up of how they approached the rollout of the game Ingress.

#6 GoPro released their GoPro Plus product early using feature flags. By breaking the larger release into smaller features with their own testing timelines, they were able to iterate and improve continuously. The video below walks through the technology they used and the timeline from dogfood to a “big bang” marketing announcement.

“At GoPro you can kind of tell we don’t things lightly. We want to do big announcements and we want to come out with great products…we actually had smaller features that would go out, and then go for alpha testing and beta testing along the way. Shortly after March, we actually had most of the applications done from a core feature standpoint, but we kept iterating and improving those core features that we knew we were going to launch with.”

 

Controlling Your Rollout Like a Boss

Did you notice some trends there? These larger companies are using beta testing to do one of the following:

  • Testing in production with feature flags
  • Ability to release early and test small functionalities before a broader release
  • Internal tests that easily become external canaries
  • Regional rollouts

As more companies start to use feature management, these incremental rollouts are not the headaches they once were. Companies can be safer and smarter with how and when they expose features to their end users.

If you want to get started with feature flagging, check out featureflags.io a resource we made for the community to learn best practices.  

08 Sep 2017

Saving Private Instances

You are a new Director of Engineering at an enterprise company. The company has just moved your product to the cloud and is hosting in Microsoft Azure. You come from a tech startup where you ran all of your software in the cloud and focused on building product. So naturally you have just implemented instrumented monitoring with HoneyComb, Kubernetes to manage your containers, and you’re thinking about leveraging serverless architecture.

You moved to this established enterprise company because they are providing technology to the healthcare industry and you are passionate about this space. They want you because you are good.

Now you face a challenge—how can I bring the good things from my past role and pair them with the security standards and compliance in my new role? Well, let’s first think about what the good things from your past role entails. Your previous team:

  • Deployed 5 times a day
  • Used feature branching
  • Feature flag first development
  • Implemented continuous integration

You realize you used 3rd party software for most of these good things, and had homegrown for others. You decide your first project is to research what software the new team is using, has built, and what software you will need to build and buy.

Security and the cloud.

Now, before we dive in, let’s consider the security standards and compliance you need to think about in your new role:

 

  • Where is my data stored?
  • What data is being sent over to the third party—is it PII?
  • Where connections are made to third parties?
  • How can I set different access control?
  • How it will work when connection fails-redundancy?
  • Is it HIPAA compliant?
  • How is all this audited?

 

Like your new company, many companies are just starting to move key operations to the cloud. This is SCARY. Moving into Azure will allow your company to free up floor space, add more server redundancy, easily scale up and down, only pay for what you use, and focus resources on revenue generating activities— security is still your responsibility.  And though these are all great things, you are not a security company.

Now you are tasked with bringing in these good processes, and they include 3rd party software providers. Products in this space are inherent in your development lifecycle and very close to your core application, but you need to ensure they are secure.

As we all know, most software providers in this space are cool tech companies in The Valley, far and isolated from the realities of your secure enterprise world. You look at on-premise options. And you remember that you’re driving change in the Healthcare space, not looking to manage infrastructure. The cloud options require a significant audit that is time consuming. A SaaS provider carries an ongoing risk that requires you store some data on 3rd party servers, which is a non-starter in Healthcare.

Is there a middle ground?

There is one more option many software providers offer: Private SaaS, also known as a managed instance, Dedicated VPC, or private instance. You surmise if they do not have on-prem, private SaaS, or a really really good security team, then it’s not likely their team is accustomed to working on enterprise challenges.

What exactly is a private SaaS offering? This refers to a dedicated single tenant, cloud software where the vendor manages the infrastructure.

Why choose a private instance?

  • Single Tenant—You will have a dedicated set of infrastructure contained within a VPC. This eliminates the risk of noisy neighbors.
  • Data Storage—Data can be stored in your AWS account or in an isolated section of the vendor’s AWS account. This allows flexibility, if you are in the EU, vendor could spin up the instance in AWS in the EU.
  • Residency—If you have a preference on where the instance is located or need to ensure proximity.
  • Compliance—If you have additional security or compliance regulations, a private instance can be customized to fit your needs.
  • Change Cadence—If you need to know when the software will be updated, you will have better insight and greater flexibility with a private instance.
  • Integrations—If you have custom tools, integrations can be built.

What’s next?

You have narrowed down a few software providers for each job you are trying to solve (continuous integration, branching, feature flagging). You understand the value, the costs, and the security implications. You have chosen to stay in the cloud, which makes your infrastructure team happy. You have chosen to use private instances and SaaS where it makes sense, which makes your security team happy. And you have the tools you need to bring the good things into your new role, so you are happy.

Now you can focus on helping your company deliver product faster, eliminate risk in your release process, speed up your product feedback loop, and do it all securely.

02 Jun 2017

Week n+1

Start? This is just the next step in the journey (image credit: Andrew Lipson)

I recently wrapped up my first official full week at LaunchDarkly. Although, I’ve been working with the team as an advisor/consultant for a number of months. Over the past year I have been advising and consulting with various start-ups looking for a good fit. I would often remark to folks that I was “company dating.”

More like introductions from friends than tinder. (Photo credit: Reddit post)

Honestly, I have very little dating experience. My wife and I met my 2nd (her 1st) year in college and haven’t looked back. Similar on the job side, I started at EMC a year after graduating and then was the first internal transfer to VMware. For almost 15 years I enjoyed the stability and resources of a large company. But, then I started to feel the need to grow and participate in the changing landscape I saw in software development trends.

Since leaving VMware I have spent a lot of time thinking about the gap between the old school development frameworks (e.g. waterfall) and newer practices (e.g. agile, scrum, continuous deployment). Tools like git, continuous integration, and automation have radically changed how we release. At EMC and VMware we measured our releases in years, or sometimes months (the same way a parent refers to their 22 month-old toddler). Compared to GitHub where we released multiple times a day.

This whole cloud thing is likely just a fad. (Photo credit: Twitter)

Recently, I’ve started to bucket these tools into three phases of for software development: Concept, Launch, and Control. I’m working on a blog series to discuss each of these in-depth, but this framework is what got me excited about LaunchDarkly. Feature management, while not the shiniest tool, provides the foundation for eliminating risk and delivering value as teams push to move faster and to be more reliable.

In addition to my passion for our product, this intelligent team, my carless commute, I did have one additional objective: to be a part of a diverse and equitable company. Not simply an organization that accepts diversity, but one that actively pursues a more diverse and inclusive team as a imperative for building better products and services. So far a great start to my next long-term relationship.

23 May 2017

Risk Elimination and The LaunchDarkly Value-Add

My first week at LaunchDarkly brought me out of the shadows in a hurry.  It began at an offsite strategy session at DFJ, our lead investor, where I learned valuable details about Waterfall vs. Agile software development methodologies. I also gained important insights into a key industry trend affecting the development community: the transition from Waterfall to Continuous Integration/Continuous Delivery (CI/CD).

What I’ve learned as a marketer has deepened my appreciation for what makes the LaunchDarkly solution so unique.

For starters, there are three main categories of customers who will benefit from partnering with LaunchDarkly:

  1. Companies interested in switching from Waterfall to CI/CD
  2. Companies currently switching/recently switched from Waterfall to CI/CD but not yet feature flagging
  3. Companies that are currently engaged in CI/CD, and using a homegrown feature management system

What’s clear is that all three of these customer segments experience different challenges. But all fall into to what our VP of Product and Platform, Adam Zimman, calls “The Risk Gap.”

What is The Risk Gap?

In software development, there is inherent risk in launching new releases. Risk in this case can be broken down into two categories:

  • Risk of losing product value
  • Risk of losing time

The longer it takes an engineering team to launch a new software release, the greater the risk of feature obsolescence. Another risk factor is competitor time to market; those companies that don’t enjoy “first mover advantage” can suffer from demoralized developers who lose interest because they can’t ship quickly enough.

The Risk Gap also means that there is greater operational risk associated with feature releases that carry greater value. The more value associated with a feature update, the greater the risk to your Ops team, because of changes made to your code base.

The Risk Gap is closely linked to the Iron Triangle concept that suggests the following:  while teams should strive to release high value features at a quick pace, the reality is that they’re often forced to pick one or the other (speed vs. quality).

The Iron Triangle mantra is “Fast, good, or cheap. Pick two.”

Let’s see how this affects the three customer categories who will benefit from using LaunchDarkly by examining the Risk Gap/Iron Triangle framework.

CategoryPainDoes Have Does Not Have
Companies interested in switching from Waterfall to CI/CDTakes Dev team a long time to launch releases.-High Quality
-Low Cost - traditional Waterfall methodology
Fast Delivery
Companies switching/recently switched from Waterfall to CI/CD but not yet feature flaggingQuality of releases is at risk.-Fast Delivery via CI/CD
-Lower Costs - not using a feature management platform
High Quality
Companies doing CI/CD + using a homegrown feature management systemA homegrown feature management system is costly to develop and maintain.-Fast Delivery - quick release cycle
-High Quality - continuous feedback loop
Lowest Cost

Each customer category is missing one of the three components of the Iron Triangle: either quality, speed, or lowest cost.

LaunchDarkly’s value-add

LaunchDarkly exists to close the Risk Gap – enabling the largest software engineering teams in the world to responsibly employ the CI/CD methodology, accelerate development cycles, eliminate the risk associated with large releases, and cut costs of developing/maintaining homegrown feature management systems.

For the first time, you don’t have to make tradeoffs with LaunchDarkly.

When you combine the great team here, a revolutionary product, and the opportunity to learn from brilliant minds every day, I am very much so looking forward to bringing our product to market.

03 May 2017

Integrating Feature Flags in Angular v4

A little while ago, we blogged about eliminating risk in your AngularJS applications by leveraging feature flags. Like all good web frameworks Angular continues to release new versions providing opportunities to tweak and update your code. The benefits of Angular over its predecessor include a built-in compiler,type enforcement, and a complete re-write in Typescript. All valuable of updates for reducing agony within the software development lifecycle.

If you’re thinking of making the switch to Angular, or are already using it, LaunchDarkly is here to help you eliminate the risk all the way from your initial migration to future successful launches. In this article, we’ll discuss how to eliminate risk and deliver value in your Angular project.

We’ll build on Tour of Heroes (which we’ll refer to as TOH from here on out), a demonstrative Angular app which showcases the framework’s basic concepts. Essentially, TOH is a live roster of superheroes, including search functionality and the ability to modify hero details. To learn more about TOH, and to get familiar with Angular, check out the official tutorial.

Creating our Feature Flags
Suppose we want to limit the usage of our search and modify features to a certain subset of our users. To achieve this, we’ll create two feature flags, toh-search  and toh-modify . In our case, we’ll allow logged in users access to search, and only the administrator will be able modify heroes.

An implementation of toh-search in the LaunchDarkly console

Integrating

Now, we’ll create a service which handles everything LaunchDarkly’s JavaScript SDK will throw at us. Note: for simplicity, we use a dummy user-switching feature (located in the user component of the project folder).

LaunchDarklyService’s constructor starts by initializing the SDK, and follows up by calling the built-in on method, which will update the feature flag values within our app whenever the user is changed, or the feature flag configurations are modified. This is handled by a Subject-typed variable,   flagChange , which will later be subscribed to by in the app’s components.
With our service functional, we’re now able inject it as a provider into TOH’s “search” and “hero” components, granting them full access to our feature flags!

In the hero-search component, we subscribe to the aforementioned flagChange , which will let Angular know that the search component should be toggled whenever the respective feature flag configuration is changed. The hero component is modified in a similar fashion to introduce the toh-modify  flag.

See it in action!

Search:

Modify:

Be sure to check out the complete project on GitHub, we’d love to see what other features you can build into Tour of Heroes!