18 Oct 2017

How We Beta Test at LaunchDarkly

Photo by Alex Holyoake on Unsplash

We recently looked at how some well-known companies beta test. Specifically we looked at groups that test in production, and do it well. As you know, testing in production is one of the best ways to find bugs and get solid feedback from your users. While some may shy away from this because of the risks involved, there are ways to mitigate risk and do it right. So this time we want to share how we beta at LaunchDarkly.

It’s no surprise that we dogfood at LaunchDarkly. Using feature flags within our development cycle is a straightforward process. We often push features directly into our production environment and safely test prior to allowing user access. When it’s time to beta test with users we can update the setting on the appropriate flag and get user feedback quickly. And of course, if we ever need to we can instantly turn features off.

Deciding Which Things to Build

When we’re thinking about new features to implement, we have our own ideas of which direction our product should go, but we also consider inbound requests. This can be from support tickets, questions from potential customers, or conversations with existing customers. Bottom line is we want to build a product that serves our customers, and so we do our best to listen to what they want.

Once we identify a feature we’d like to build—whether it was our own idea or a customer request—we’ll share it out to see if other customers are also interested. This is an important part of our beta testing process, because once the feature is in production and we’re ready to test it, these are the people we want to circle back with for beta testing.

Testing in Production

When it’s time to test, we test with actual end users in production. Our feature management platform allows us to turn features on for specific users. We can specify individual users, or we can expose users by attribute, like region (everyone in Denver)—and we can instantly turn them off at any time.

Because we’re testing in production, we don’t have to have an isolated environment or separate account. For those customers who showed interest, and agreed to participate in beta testing, we turn the features on in their production accounts.

Typically we beta for two weeks, sometimes as long as a month. As mentioned before, since we know which customers are interested in the feature, we can go back to them and have them test it. These are the users who already know they want this functionality, so we want to be sure it fits (or exceeds!) their expectations. And of course we want to make the most of this time, so it’s important we actually get feedback. We find that those who have asked for the feature are eager to let us know how things are working. We make a point of also following up with those who don’t proactively offer feedback—we want to hear from everyone!

While we’re testing and getting feedback, we’re taking all this information in and improving the feature before rolling it out to everyone else. When we feel confident we have something that’s ready to be shared, we’ll begin a percentage rollout to the rest of our users.

Embracing Failure

Using feature flags around features within our development cycles allows us to mitigate risk by pushing out small, incremental changes at a time. As you can see, this also enables us to beta test quickly and safely. If there are major bugs, we’re more likely to identify them early on before affecting our all of our customers.

“Embrace failure. Chaos and failure are your friends. The issue is not if you will fail, it is when you will fail, and whether you will notice.” -Charity Majors

Right now we’re currently in beta for scoped access tokens and a new faster .net SDK. Let us know if you’d like to take a look at it early, we’d love to hear what you think.

02 Oct 2017

Removing Risk from Product Launches: A webinar with LaunchDarkly, CircleCI, and GoPro

We recently sat down with one of GoPro’s Senior Engineering Managers, Andrew Maxwell, and the CTO of CircleCI to discuss reducing risk in product launches. Andrew talked about how his team delivered their code two weeks early, tested in production, and had an overall successful product launch. He goes into detail, sharing how his team uses continuous integration and feature flags to make product launches like that possible.

The Big Launch

Andrew’s team is responsible for web applications. Last September his team was focused on a big initiative around a product launch called GoPro Plus, which allows users to access and share content wherever they are. This launch included both mobile and desktop apps, and promoted two new cameras in the GoPro line.  

Two Weeks

The team delivered their code and pushed it into production two weeks early:

“…we used LaunchDarkly to push our code to production, turn the apps off — off by default—and then make sure that we had everything pushed out, deployed, and the infrastructure running live without customers actually seeing it.”

Two weeks gave them time to do a full integration test their features. They tested both in-house and in production—slowly opening up who had access—so they could get valuable feedback, find bugs, and make continuous improvements in the weeks leading up to the big launch. On the day-of, they were confident in their work and simply turned on 12 feature flags.

Check out the full webinar below to learn more about how the team used CircleCI and LaunchDarkly, their planning strategies, and best practices for their continuous integration pipeline.