02 Apr 2018

Targeting Users with Feature Flags

Companies exploring feature management are looking for control over their releases. A common theme is using feature flags to rollout a feature to a small percentage of users, or quickly roll a feature back if it is not working properly. These future flaggers also seek to control a feature further by limiting its visibility to individual users or a group of users. In this piece, we’ll explore how LaunchDarkly lets you control your releases in these ways.

When launching a new feature, or simply updating an existing feature, you may not always want everyone to receive the same experience. When working with a new front-end marketing design, a back-end improvement to search, or anything in between, targeting specific users can help ensure the best experience for all of your customers. Below are a few common scenarios where targeting can come into play. We’d love to hear your thoughts on other targeting best practices—let us know what you come up with!

Targeting users within LaunchDarkly’s UI.

Internal QA

Perhaps the most common use case for individual user targeting is internal QA. Testing in production is a scary thought, but so is launching to production without knowing how the rollout will go. Feature flags enable safe testing in production. By targeting only your internal QA teams to receive the new feature, you can experience how it will function in a production environment without exposing the untested feature to your customers.

Beta Releases

When releasing a feature or product in beta, you can target users in your beta group to receive this update while not releasing it to anyone else. Essentially, you would set the flag to “true” for all beta users to enable the new feature. Everyone else would have the flag set to “false”. When Kevin from LaunchDarkly asks to be part of your beta group, it’s as easy as adding that user to the true variation to enable the new feature.

Attribute-Based Targeting

Sometimes, you’ll want to target users, but may not want to dive into as many specifics as we just visited. Perhaps you are constrained by regulations, and need to deliver a different experience to customers from different states. Or, maybe your marketing team is launching a campaign specifically to Gmail users. You can place the email domain “gmail.com” in your targeting rules to deliver an experience only to that group.

Targeting in a Canary Launch

Many companies are seeking to slowly rollout their releases to before releasing to everybody. Rolling a new feature out to 10% of users, for example, is a great way to gain validation before a complete launch. You can target your internal QA team to be a part of the 10% initial rollout, or specifically include members from a beta group to see the new feature first. Conversely, you can target specific users or groups to exclude from the launch, and give them the new feature only when it is rolled out to everybody.

A common challenge with canary launches is delivering a consistent experience to users from the same group, especially in B2B applications. LaunchDarkly’s platform provides a bucketing feature that solves this problem. By bucketing users, you are ensuring that all users from the same group receive the same variation for the duration of a canary launch.

Other Considerations

The examples we have covered so far are typically used for short-term flags, either upon initial rollout or for a shorter-lived feature. As your company’s feature flag use continues to mature, wrapping a feature or set of features in a long-term or permanent flag is a logical next step. A common use case here applies to entitlements. For example, if you have Basic, Pro, and Elite versions of your product, user targeting helps manage these tiers effectively across your organization. Many products with different account tiers already have a way of controlling functionality, but with a feature management platform, this can be easily managed by anyone on your team. Customer success team members can quickly dial functionality up or down without the need to go to engineering for support. When a customer upgrades from Pro to Elite, a user simply flips the appropriate flags on.

Managing targeting rules at scale is an important consideration, especially as the amount of feature flags you are managing continues to grow. We introduced a new feature we call Segments to help tackle this very topic. Segments enables you to build reusable targeting rules to make this practice quick and scalable. Borrowing from a few of my examples above, you may want to flip a feature on for everyone with an Elite subscription, but also include all beta members as well. Because this is an important feature, you may want to also target internal users to ensure rollouts go smoothly. This same rollout strategy will likely apply to future features and you can simply target your Segment to receive the true or false variation. We’ll be featuring Segments in a future post, so stay tuned for more.

Build reusable targeting rules with Segments

Data Privacy and Targeting

Targeting often involves customer data, and we launched Private Attributes in January to add an additional layer of protection. Private Attributes allows you to shield your customer data from LaunchDarkly while still enabling targeting rules on the shielded data. One thing I’d like to point out is that you never need to upload a database of users into LaunchDarkly to enable targeting. LaunchDarkly simply stores your targeting rules, which are evaluated based on the user objects that you provide. Ben Nadel wrote a wonderful piece describing InVision’s use of targeting and Private Attributes: Using LaunchDarkly To Target Personally Identifiable Information (PII) During Feature Flag Evaluation Without Leaking Sensitive Data.

To wrap things up, there are a number of different ways targeting users can deliver a more valuable experience. We visited just a few above, but the opportunities here are endless. At the end of the day, you’ll target users based on what makes the most sense for your product. We’d love to hear your thoughts and how you think user targeting can help. Until then and for more pointers, you can visit our documentation here.

18 Oct 2017

How We Beta Test at LaunchDarkly

Photo by Alex Holyoake on Unsplash

We recently looked at how some well-known companies beta test. Specifically we looked at groups that test in production, and do it well. As you know, testing in production is one of the best ways to find bugs and get solid feedback from your users. While some may shy away from this because of the risks involved, there are ways to mitigate risk and do it right. So this time we want to share how we beta at LaunchDarkly.

It’s no surprise that we dogfood at LaunchDarkly. Using feature flags within our development cycle is a straightforward process. We often push features directly into our production environment and safely test prior to allowing user access. When it’s time to beta test with users we can update the setting on the appropriate flag and get user feedback quickly. And of course, if we ever need to we can instantly turn features off.

Deciding Which Things to Build

When we’re thinking about new features to implement, we have our own ideas of which direction our product should go, but we also consider inbound requests. This can be from support tickets, questions from potential customers, or conversations with existing customers. Bottom line is we want to build a product that serves our customers, and so we do our best to listen to what they want.

Once we identify a feature we’d like to build—whether it was our own idea or a customer request—we’ll share it out to see if other customers are also interested. This is an important part of our beta testing process, because once the feature is in production and we’re ready to test it, these are the people we want to circle back with for beta testing.

Testing in Production

When it’s time to test, we test with actual end users in production. Our feature management platform allows us to turn features on for specific users. We can specify individual users, or we can expose users by attribute, like region (everyone in Denver)—and we can instantly turn them off at any time.

Because we’re testing in production, we don’t have to have an isolated environment or separate account. For those customers who showed interest, and agreed to participate in beta testing, we turn the features on in their production accounts.

Typically we beta for two weeks, sometimes as long as a month. As mentioned before, since we know which customers are interested in the feature, we can go back to them and have them test it. These are the users who already know they want this functionality, so we want to be sure it fits (or exceeds!) their expectations. And of course we want to make the most of this time, so it’s important we actually get feedback. We find that those who have asked for the feature are eager to let us know how things are working. We make a point of also following up with those who don’t proactively offer feedback—we want to hear from everyone!

While we’re testing and getting feedback, we’re taking all this information in and improving the feature before rolling it out to everyone else. When we feel confident we have something that’s ready to be shared, we’ll begin a percentage rollout to the rest of our users.

Embracing Failure

Using feature flags around features within our development cycles allows us to mitigate risk by pushing out small, incremental changes at a time. As you can see, this also enables us to beta test quickly and safely. If there are major bugs, we’re more likely to identify them early on before affecting our all of our customers.

“Embrace failure. Chaos and failure are your friends. The issue is not if you will fail, it is when you will fail, and whether you will notice.” -Charity Majors

Right now we’re currently in beta for scoped access tokens and a new faster .net SDK. Let us know if you’d like to take a look at it early, we’d love to hear what you think.

14 Sep 2017

Beta Testing with Feature Toggles: Testing in Production Like a Pro

We all know beta testing is important—not just for understanding your customers’ needs, but also for stability and security. Every time you do a launch you are essentially asking: “Are there bugs? Is there feedback?” Both with the goal of making your product better.

Testing in production will give you the most information about the success of your new functionality. And because feature flags help separate deployment from release, they make such testing safe and easy. When it comes to beta testing, a lot of the top companies tend to adhere to a similar paradigm—test early, test often, and do it in your production environment.

So how do companies have smooth and simple transitions from alpha to beta testing, and then to full rollout? Read on to learn how top companies are approaching their beta testing using deployment tools with feature flags providing links out to more in-depth descriptions.

But before we get started, here’s a quick terminology review. Pete Hodgson refers to this use of feature flags for betas as “permissioning toggles.” Also known as a “canary launch,” this is often random like a percentage rollout. A set group, or “champagne brunch,” releases to internal users or another section or group.


6 Approaches to Product Launching

#1 Facebook is the prime example of dark launching. Their release management has to be impeccable to operate at such massive scale. Their betas are often up to  million users or more.

“Although we push to production only once a week, it’s still important to test the code early in real-world settings so that engineers can get quick feedback. We make mobile release candidates available every day for canary users, including 1 million or so Android beta testers.”

Read their article on Rapid Release at Massive Scale to learn more about how they do continuous delivery at scale.

#2 Hootsuite gives a typical rollout pattern for its features—starting internally and then slowly exposing to a larger audience.

Typical
Push new code then:
– Dark launch to yourself or your team to test
– Launch to the whole Hootsuite organization
– 10% of all users
– Watch graphs
– 50%
– 100%
– Simple means of rollback if necessary

Check out Bill Monkman’s full deck on dark launching here.

#3 Etsy calls feature flags “config flags,” and gives a lot of credit for their process to Flickr.

“Key system-level and business level metrics (like checkout/listing/registration/sign-in rates) are projected on screens in the office and we have a number of internal dashboards that the team uses (we mainly use Ganglia and Graphite). We also have lots of switches and knobs to help us roll features out to percentages of users and ramp them up slowly, or quickly. Features are used and tested by us here at Etsy for some period of time before they are rolled out publicly.”

They have custom built a feature flagging API, “Feature API” to enable this. Some of the bucketing they use include: admin, internal, users, groups.Read more about Etsy’s deployment practices and check out their Feature API on GitHub.

#4 Beta can also apply to back-end rollouts. Instagram does canary deployments to a subset of servers, using feature flags as a continuous delivery tool. It’s important for continuous delivery to perform these tests, which are key in helping them avoid failed deployments.

But Instagram hasn’t always had this system. Read here to learn how they evolved from a “mish-mash of manual steps and scripts” to a system they could depend on. And check this out if you want more recipes for database migration with feature flags.

#5 Niantic’s Pokemon Go betas are well known and rabidly tracked by its fans. They famously roll out by region—a field test in Japan here, a limited beta in Australia, and then something in New Zealand. Sometimes these betas for features are invite-only. Here’s a write up of how they approached the rollout of the game Ingress.

#6 GoPro released their GoPro Plus product early using feature flags. By breaking the larger release into smaller features with their own testing timelines, they were able to iterate and improve continuously. The video below walks through the technology they used and the timeline from dogfood to a “big bang” marketing announcement.

“At GoPro you can kind of tell we don’t things lightly. We want to do big announcements and we want to come out with great products…we actually had smaller features that would go out, and then go for alpha testing and beta testing along the way. Shortly after March, we actually had most of the applications done from a core feature standpoint, but we kept iterating and improving those core features that we knew we were going to launch with.”

 

Controlling Your Rollout Like a Boss

Did you notice some trends there? These larger companies are using beta testing to do one of the following:

  • Testing in production with feature flags
  • Ability to release early and test small functionalities before a broader release
  • Internal tests that easily become external canaries
  • Regional rollouts

As more companies start to use feature management, these incremental rollouts are not the headaches they once were. Companies can be safer and smarter with how and when they expose features to their end users.

If you want to get started with feature flagging, check out featureflags.io a resource we made for the community to learn best practices.  

22 Dec 2016

Beta Testing using Feature Flags

LaunchDarkly Feature Toggle Beta

It’s best practice for products to have some sort of beta – a way to collect customer feedback and test performance before releasing to everyone. In an era of continuous delivery, we are delivering new features and experiences more frequently and with less time to gather thorough customer and performance feedback. With this increased cadence, product teams are having to make betas shorter, forego them altogether, or slow down their release cadence to gather adequate customer feedback.

Challenges of traditional betas:

  • Coordinating Opt-Ins: It sometimes takes weeks or months to gather customer opt-ins to test new betas. You also have to organize the distribution of beta keys (ex. for early access to games) and reminder emails.
  • Organizing Focus Groups: Getting feedback from focus groups is often time consuming and expensive, creating a long feedback loop that lengthens the release process.
  • Opt-Out: If customers opt-in to a beta and don’t like the experience, then they will want a simple mechanism to switch back to the production version.
  • Granular Betas: It is very difficult to do targeted betas based on user attributes or to perform incremental percentage rollouts of new beta features.

Feature toggles

To overcome these challenges, smart product teams are beginning to run betas with feature flags/toggles. These are mechanisms for granularly controlling software releases, allowing you to control the timing and visibility of a beta release.

Currently, many betas are tied to code releases and are managed by a config file or database.  This approach requires engineering time or custom mechanisms to opt-in users.

With feature toggles, you can empower product, marketing, sales, and even customers (themselves) to opt-in new to a new beta experience.

Feature Toggle Beta Test LaunchDarkly

In this simple example, you can use a toggle to control the visibility of a new beta feature. Ideally, this toggle would be part of a user interface that could be controlled by a non-technical team member. The code, itself, could be deployed off and then turned on via the toggle.

Beta Test Percentage Rollout with Feature Toggle LaunchDarkly

Moreover, you can also use the toggle to control the percentage of users who get the beta experience. For instance, you could release the new beta experience and have it rolled out to 0% of users. You could gradually increase the rollout percentage from 1% to 5% to 20% and more, collecting customer and performance feedback along the way.

Surfacing this beta control functionality in a user interface is critical for giving non-technical team members access to release controls.

Regional betas

For a recent example of a targeted rollout, we can look at how Pokémon GO released their product country by country: first to the United States and then abroad.

This is a great use case for feature toggles because you can create targeting rules to determine which users receive the feature first. For example, I could create a toggle that is governed by the rule: “If users live in San Francisco, then serve the new Nearby Pokémon feature”. This allows you to maintain different regional feature sets without having to deploy different versions of the application. It also allowed Pokémon GO to refine their algorithms and assess customer feedback before rolling out the new feature to a wider audience.

Benefits of beta testing with feature toggles

  • Empowered non-technical users: Allow the sales, marketing, product, design, and business teams to turn features on for specific users, collect feedback, and control the business logic. This also substantially cuts down on engineering time.
  • Production feedback for your beta tests: Test features in production with limited user segments to collect customer and performance feedback.
  • Incremental percentage rollouts: Gradually roll out features to incrementally test performance and mitigate risk. If the feature is bad, toggle it off.
  • Real-time opt-in / opt-out: Allow users to opt in and out of beta tests in real time, controlled via the feature toggle. Skylight provides a nice article on this.

Getting started with toggles

Conceptually, a feature toggle is relatively simple. You create a conditional in your code that controls the visibility of a code snippet. There are many open source libraries that will allow you to get started.  However, these libraries become cumbersome when you try to feature toggle at scale or restrict access to particular toggles. Depending on your needs, you could consider a feature toggle management platform to provide a system for access control and mitigating technical debt.

19 May 2016

Case study: Modernizing development & controlling rollout with LaunchDarkly

The challenge

The company, a leading online real estate marketplace, sought to bring its development up to date with modern processes. To accomplish this transformation, they hired management with experience at the most idolized mega tech companies.  The first thing on the manager’s agenda? Move faster, with less risk by using a feature flagging system like the one they had built at their previous company.  

Just for some background – at this juncture, the team was doing three-week sprint cycles. They had feature branches that they would develop every three weeks and then push out. They would deploy five features at once and if anything went wrong, they’d have to roll it all back.  

Continue reading “Case study: Modernizing development & controlling rollout with LaunchDarkly” »